* (unknown), @ 2009-05-07 17:01 Bevan Watkiss 2009-05-07 17:13 ` Alex Riesen 0 siblings, 1 reply; 34+ messages in thread From: Bevan Watkiss @ 2009-05-07 17:01 UTC (permalink / raw) To: git I am trying to create a working tree for people to read from and have it update from a bare repository regularly. Right now I am using git-pull to fetch the changes, but its running slow due to the size of my repo and the speed of the hardware as it seems to be checking the working tree for any changes. Is there a way to make the pull ignore the local working tree and only look at files that are changed in the change sets being pulled? Bevan ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-07 17:01 (unknown), Bevan Watkiss @ 2009-05-07 17:13 ` Alex Riesen 2009-05-07 17:26 ` Bevan Watkiss 0 siblings, 1 reply; 34+ messages in thread From: Alex Riesen @ 2009-05-07 17:13 UTC (permalink / raw) To: Bevan Watkiss; +Cc: git 2009/5/7 Bevan Watkiss <bevan.watkiss@cloakware.com>: > I am trying to create a working tree for people to read from and have it > update from a bare repository regularly. Right now I am using git-pull to > fetch the changes, but it’s running slow due to the size of my repo and the > speed of the hardware as it seems to be checking the working tree for any > changes. > > Is there a way to make the pull ignore the local working tree and only look > at files that are changed in the change sets being pulled? Assuming you didn't modify that directory you pull into, git pull will do almost exactly what you described. Almost, because the operation (the merge) will involve looking for local changes (committed and not). It should be faster to do something like this: git fetch && git reset --hard origin/master Again, assuming the directory supposed to be read-only. Otherwise, you have to merge (i.e. git pull). ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 17:13 ` Alex Riesen @ 2009-05-07 17:26 ` Bevan Watkiss 2009-05-07 18:18 ` Alex Riesen 2009-05-07 18:56 ` Linus Torvalds 0 siblings, 2 replies; 34+ messages in thread From: Bevan Watkiss @ 2009-05-07 17:26 UTC (permalink / raw) To: 'Alex Riesen'; +Cc: git It's the looking for local changes I'm trying to avoid. Doing a reset still goes over the tree, which isn't helpful. Basically I have a copy of my tree where only git can write to it, so I know the files are right. The NAS box I have the tree on is slow, so reading the tree adds about 10 minutes to the process when I only want to update a few files. -----Original Message----- From: Alex Riesen [mailto:raa.lkml@gmail.com] Sent: May 7, 2009 1:14 PM To: Bevan Watkiss Cc: git@vger.kernel.org Subject: Re: 2009/5/7 Bevan Watkiss <bevan.watkiss@cloakware.com>: > I am trying to create a working tree for people to read from and have it > update from a bare repository regularly. Right now I am using git-pull to > fetch the changes, but its running slow due to the size of my repo and the > speed of the hardware as it seems to be checking the working tree for any > changes. > > Is there a way to make the pull ignore the local working tree and only look > at files that are changed in the change sets being pulled? Assuming you didn't modify that directory you pull into, git pull will do almost exactly what you described. Almost, because the operation (the merge) will involve looking for local changes (committed and not). It should be faster to do something like this: git fetch && git reset --hard origin/master Again, assuming the directory supposed to be read-only. Otherwise, you have to merge (i.e. git pull). ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-07 17:26 ` Bevan Watkiss @ 2009-05-07 18:18 ` Alex Riesen 2009-05-07 18:48 ` Bevan Watkiss 2009-05-07 18:56 ` Linus Torvalds 1 sibling, 1 reply; 34+ messages in thread From: Alex Riesen @ 2009-05-07 18:18 UTC (permalink / raw) To: Bevan Watkiss; +Cc: git 2009/5/7 Bevan Watkiss <bevan.watkiss@cloakware.com>: > It's the looking for local changes I'm trying to avoid. Doing a reset still > goes over the tree, which isn't helpful. The stat(2) is slow? Then try setting core.ignoreStat (see manpage of git config) to true: git config core.ignorestat true and read below. > Basically I have a copy of my tree where only git can write to it, so I know > the files are right. The NAS box I have the tree on is slow, so reading the > tree adds about 10 minutes to the process when I only want to update a few > files. Try "git checkout origin/master". It uses index and shouldn't checkout files which are uptodate with the index. And actually, git merge should fast-forward, in your case and will update just the changed files... Of course, you can always compare HEAD and origin/master, and resolve the changes yourself (see git diff -z --name-status), but it is unlikely to be any faster. ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 18:18 ` Alex Riesen @ 2009-05-07 18:48 ` Bevan Watkiss 2009-05-07 19:56 ` Björn Steinbrink 0 siblings, 1 reply; 34+ messages in thread From: Bevan Watkiss @ 2009-05-07 18:48 UTC (permalink / raw) To: 'Alex Riesen'; +Cc: git Still took 11 minutes. The idea I've come up with today is something along the lines of git fetch origin/master git log --name-only ..<hash> | xargs git checkout -f -- This should work to quickly keep my files upto date, and I can then periodically pull properly to move the HEAD. Thanks for the info Bevan -----Original Message----- From: Alex Riesen [mailto:raa.lkml@gmail.com] Sent: May 7, 2009 2:18 PM To: Bevan Watkiss Cc: git@vger.kernel.org Subject: Re: 2009/5/7 Bevan Watkiss <bevan.watkiss@cloakware.com>: > It's the looking for local changes I'm trying to avoid. Doing a reset still > goes over the tree, which isn't helpful. The stat(2) is slow? Then try setting core.ignoreStat (see manpage of git config) to true: git config core.ignorestat true and read below. > Basically I have a copy of my tree where only git can write to it, so I know > the files are right. The NAS box I have the tree on is slow, so reading the > tree adds about 10 minutes to the process when I only want to update a few > files. Try "git checkout origin/master". It uses index and shouldn't checkout files which are uptodate with the index. And actually, git merge should fast-forward, in your case and will update just the changed files... Of course, you can always compare HEAD and origin/master, and resolve the changes yourself (see git diff -z --name-status), but it is unlikely to be any faster. ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-07 18:48 ` Bevan Watkiss @ 2009-05-07 19:56 ` Björn Steinbrink 0 siblings, 0 replies; 34+ messages in thread From: Björn Steinbrink @ 2009-05-07 19:56 UTC (permalink / raw) To: Bevan Watkiss; +Cc: 'Alex Riesen', git [Please don't top-post...] On 2009.05.07 14:48:20 -0400, Bevan Watkiss wrote: > From: Alex Riesen [mailto:raa.lkml@gmail.com] > > 2009/5/7 Bevan Watkiss <bevan.watkiss@cloakware.com>: > > > It's the looking for local changes I'm trying to avoid. Doing a > > > reset still goes over the tree, which isn't helpful. > > > > The stat(2) is slow? Then try setting core.ignoreStat (see manpage > > of git config) to true: git config core.ignorestat true and read > > below. > > Still took 11 minutes. IIRC, to see the effects of core.ignorestat, you need to have updated all files once. So you might need, for example, "git checkout -f HEAD" (not sure if a plain checkout is enough) once first, and then the future "git checkout $something" should be faster. Björn ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 17:26 ` Bevan Watkiss 2009-05-07 18:18 ` Alex Riesen @ 2009-05-07 18:56 ` Linus Torvalds 2009-05-07 19:37 ` RE: Bevan Watkiss 1 sibling, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 18:56 UTC (permalink / raw) To: Bevan Watkiss; +Cc: 'Alex Riesen', git On Thu, 7 May 2009, Bevan Watkiss wrote: > > Basically I have a copy of my tree where only git can write to it, so I know > the files are right. The NAS box I have the tree on is slow, so reading the > tree adds about 10 minutes to the process when I only want to update a few > files. Ouch. You could try doing [core] preloadindex = true and see if that helps some of your loads. It does limit even the parallel tree stat to 20 or so, but if most of your cost is in just doing the lstat() over the files to see that they haven't changed, you might be getting a factor-of-20 speedup for at least _some_ of what you do. If you can, it might also be interesting to see system call trace patterns (with times!) to see if there is something obviously horribly bad going on. If you're running under Linux, and don't think the data contains anything very private, send me the output of "strace -f -T" of the most problematic operations, and maybe I can see if I can come up with anything interesting. I have long refused to use networked filesystems because I used to find them -so- painful when working with CVS, so none of my performance work has ever really directly concentrated on long-latency filesystems. Even the index preload was all done "blind" with other people reporting issues (and happily I could see some of the effects with local filesystems and multiple CPU's ;). Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 18:56 ` Linus Torvalds @ 2009-05-07 19:37 ` Bevan Watkiss 2009-05-07 20:07 ` RE: Linus Torvalds 0 siblings, 1 reply; 34+ messages in thread From: Bevan Watkiss @ 2009-05-07 19:37 UTC (permalink / raw) To: 'Linus Torvalds'; +Cc: 'Alex Riesen', git Looking at the trace it does appear that most of this is the lstat. It's the problem of having many tiny files on a network drive, and trying to use git for something it's not meant. The log has 265430 lines of lstat and 10887 other lines. If you still want the log file I'll strip out the directory names and send it off. It would be nice to have an option that you can pull only the files that changed in the changesets you are updating and ignore the state of the other files. Bevan -----Original Message----- From: Linus Torvalds [mailto:torvalds@linux-foundation.org] Sent: May 7, 2009 2:56 PM To: Bevan Watkiss Cc: 'Alex Riesen'; git@vger.kernel.org Subject: RE: On Thu, 7 May 2009, Bevan Watkiss wrote: > > Basically I have a copy of my tree where only git can write to it, so I know > the files are right. The NAS box I have the tree on is slow, so reading the > tree adds about 10 minutes to the process when I only want to update a few > files. Ouch. You could try doing [core] preloadindex = true and see if that helps some of your loads. It does limit even the parallel tree stat to 20 or so, but if most of your cost is in just doing the lstat() over the files to see that they haven't changed, you might be getting a factor-of-20 speedup for at least _some_ of what you do. If you can, it might also be interesting to see system call trace patterns (with times!) to see if there is something obviously horribly bad going on. If you're running under Linux, and don't think the data contains anything very private, send me the output of "strace -f -T" of the most problematic operations, and maybe I can see if I can come up with anything interesting. I have long refused to use networked filesystems because I used to find them -so- painful when working with CVS, so none of my performance work has ever really directly concentrated on long-latency filesystems. Even the index preload was all done "blind" with other people reporting issues (and happily I could see some of the effects with local filesystems and multiple CPU's ;). Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 19:37 ` RE: Bevan Watkiss @ 2009-05-07 20:07 ` Linus Torvalds 2009-05-07 20:20 ` RE: Linus Torvalds 0 siblings, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 20:07 UTC (permalink / raw) To: Bevan Watkiss; +Cc: 'Alex Riesen', git On Thu, 7 May 2009, Bevan Watkiss wrote: > > Looking at the trace it does appear that most of this is the lstat. It's > the problem of having many tiny files on a network drive, and trying to use > git for something it's not meant. > > The log has 265430 lines of lstat and 10887 other lines. If you still want > the log file I'll strip out the directory names and send it off. Actually, if it's just the lstat's, then it's not all that interesting any more, it's a known problem with at least a known _partial_ solution. However, I think it turns out that we've only enabled the index preloading with "git diff" and "git commit". Not on "git checkout". So start off doing that > [core] > preloadindex = true AND apply the following patch to git, and see how much (if any) that helps. It sounds like you have a pretty damn large repository, together with a slow filesystem. It really could be a big improvement. The patch is TOTALLY UNTESTED. It also worries me that 'git checkout' seems to do _two_ 'lstat()' calls per file. I didn't look any more closely, but there may be other issues here. Linus --- builtin-checkout.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/builtin-checkout.c b/builtin-checkout.c index 15f0c32..3100ccd 100644 --- a/builtin-checkout.c +++ b/builtin-checkout.c @@ -216,7 +216,7 @@ static int checkout_paths(struct tree *source_tree, const char **pathspec, struct lock_file *lock_file = xcalloc(1, sizeof(struct lock_file)); newfd = hold_locked_index(lock_file, 1); - if (read_cache() < 0) + if (read_cache_preload(pathspec) < 0) return error("corrupt index file"); if (source_tree) @@ -367,7 +367,7 @@ static int merge_working_tree(struct checkout_opts *opts, int newfd = hold_locked_index(lock_file, 1); int reprime_cache_tree = 0; - if (read_cache() < 0) + if (read_cache_preload(NULL) < 0) return error("corrupt index file"); cache_tree_free(&active_cache_tree); ^ permalink raw reply related [flat|nested] 34+ messages in thread
* RE: 2009-05-07 20:07 ` RE: Linus Torvalds @ 2009-05-07 20:20 ` Linus Torvalds 2009-05-07 20:43 ` Junio C Hamano 2009-05-07 21:55 ` Linus Torvalds 0 siblings, 2 replies; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 20:20 UTC (permalink / raw) To: Bevan Watkiss; +Cc: 'Alex Riesen', git On Thu, 7 May 2009, Linus Torvalds wrote: > > The patch is TOTALLY UNTESTED. It also worries me that 'git checkout' > seems to do _two_ 'lstat()' calls per file. I didn't look any more > closely, but there may be other issues here. Hmm. The second pass comes from show_local_changes(&new->commit->object); (this is the "git checkout" without actual filenames), and is suppressed if we ask for a quiet checkout. But it's sad how it re-loads the index. I wonder where the CE_VALID bit got dropped. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-07 20:20 ` RE: Linus Torvalds @ 2009-05-07 20:43 ` Junio C Hamano 2009-05-07 21:33 ` Re: Linus Torvalds 2009-05-07 21:55 ` Linus Torvalds 1 sibling, 1 reply; 34+ messages in thread From: Junio C Hamano @ 2009-05-07 20:43 UTC (permalink / raw) To: Linus Torvalds; +Cc: Bevan Watkiss, 'Alex Riesen', git Linus Torvalds <torvalds@linux-foundation.org> writes: > On Thu, 7 May 2009, Linus Torvalds wrote: >> >> The patch is TOTALLY UNTESTED. It also worries me that 'git checkout' >> seems to do _two_ 'lstat()' calls per file. I didn't look any more >> closely, but there may be other issues here. > > Hmm. The second pass comes from > > show_local_changes(&new->commit->object); > > (this is the "git checkout" without actual filenames), and is suppressed > if we ask for a quiet checkout. But it's sad how it re-loads the index. I > wonder where the CE_VALID bit got dropped. I do not think you mean CE_VALID; CE_UPTODATE isn't it? ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-07 20:43 ` Junio C Hamano @ 2009-05-07 21:33 ` Linus Torvalds 0 siblings, 0 replies; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 21:33 UTC (permalink / raw) To: Junio C Hamano; +Cc: Bevan Watkiss, 'Alex Riesen', git On Thu, 7 May 2009, Junio C Hamano wrote: > > I do not think you mean CE_VALID; CE_UPTODATE isn't it? Yes, sorry. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 20:20 ` RE: Linus Torvalds 2009-05-07 20:43 ` Junio C Hamano @ 2009-05-07 21:55 ` Linus Torvalds 2009-05-07 22:27 ` RE: david ` (2 more replies) 1 sibling, 3 replies; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 21:55 UTC (permalink / raw) To: Bevan Watkiss; +Cc: 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, Linus Torvalds wrote: > > Hmm. The second pass comes from > > show_local_changes(&new->commit->object); > > (this is the "git checkout" without actual filenames), and is suppressed > if we ask for a quiet checkout. But it's sad how it re-loads the index. I > wonder where the CE_VALID bit got dropped. Ahh. It's not actually dropped, it's still there. It's just that 'get_stat_data()' doesn't check it, when asking for noncached data. The logic of 'get_stat_data()' is that it will return the stat data from the filesystem (unless we explicitly ask for just the cached case, in which case it will take it from the cache entry directly). However, the code doesn't realize that if ce_uptodate() is true, then we already know the stat data, so no need to do the lstat() again, and we can take it all from the cache entry regardless of whether we asked for filesystem data or cached data. So here's a better patch. It should cut down the 'lstat()' calls from "git checkout" a lot. It looks obvious enough, and it passes testing (and now "git checkout" only does about as many lstat's as there are files in the repository, and they seem to all be properly asynchronous if 'core.preloadindex' is set. Somebody should check. It would be interesting to hear about whether this makes a performance impact, especially with slow filesystems and/or other operating systems that have a relatively higher cost for 'lstat()'. Linus --- builtin-checkout.c | 4 ++-- diff-lib.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/builtin-checkout.c b/builtin-checkout.c index 15f0c32..3100ccd 100644 --- a/builtin-checkout.c +++ b/builtin-checkout.c @@ -216,7 +216,7 @@ static int checkout_paths(struct tree *source_tree, const char **pathspec, struct lock_file *lock_file = xcalloc(1, sizeof(struct lock_file)); newfd = hold_locked_index(lock_file, 1); - if (read_cache() < 0) + if (read_cache_preload(pathspec) < 0) return error("corrupt index file"); if (source_tree) @@ -367,7 +367,7 @@ static int merge_working_tree(struct checkout_opts *opts, int newfd = hold_locked_index(lock_file, 1); int reprime_cache_tree = 0; - if (read_cache() < 0) + if (read_cache_preload(NULL) < 0) return error("corrupt index file"); cache_tree_free(&active_cache_tree); diff --git a/diff-lib.c b/diff-lib.c index a310fb2..0aba6cd 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -214,7 +214,7 @@ static int get_stat_data(struct cache_entry *ce, const unsigned char *sha1 = ce->sha1; unsigned int mode = ce->ce_mode; - if (!cached) { + if (!cached && !ce_uptodate(ce)) { int changed; struct stat st; changed = check_removed(ce, &st); ^ permalink raw reply related [flat|nested] 34+ messages in thread
* RE: 2009-05-07 21:55 ` Linus Torvalds @ 2009-05-07 22:27 ` david 2009-05-07 22:36 ` RE: Linus Torvalds 2009-05-08 8:17 ` Alex Riesen 2009-05-08 16:47 ` 'git checkout' and unlink() calls (was: Re: ) Kjetil Barvik 2 siblings, 1 reply; 34+ messages in thread From: david @ 2009-05-07 22:27 UTC (permalink / raw) To: Linus Torvalds; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, Linus Torvalds wrote: this patch is worthwhile in itself, but the use case that is presented here is slightly different, and I wonder if it's common enough to be worth having a config option for. his use case (as I understand it) is that the working tree is never updated by anything other than git. it never recieves patches or manual edits. as such _any_ lstats of the tree are a waste of time. if git knows what was checked out before and what is being checked out now, it can find what files need to be changed. this situation is not common for most developers, but it would be reasonable for build farms, so it's not just a one-person issue. David Lang > On Thu, 7 May 2009, Linus Torvalds wrote: >> >> Hmm. The second pass comes from >> >> show_local_changes(&new->commit->object); >> >> (this is the "git checkout" without actual filenames), and is suppressed >> if we ask for a quiet checkout. But it's sad how it re-loads the index. I >> wonder where the CE_VALID bit got dropped. > > Ahh. It's not actually dropped, it's still there. > > It's just that 'get_stat_data()' doesn't check it, when asking for > noncached data. > > The logic of 'get_stat_data()' is that it will return the stat data from > the filesystem (unless we explicitly ask for just the cached case, in > which case it will take it from the cache entry directly). > > However, the code doesn't realize that if ce_uptodate() is true, then we > already know the stat data, so no need to do the lstat() again, and we > can take it all from the cache entry regardless of whether we asked for > filesystem data or cached data. > > So here's a better patch. It should cut down the 'lstat()' calls from "git > checkout" a lot. > > It looks obvious enough, and it passes testing (and now "git checkout" > only does about as many lstat's as there are files in the repository, and > they seem to all be properly asynchronous if 'core.preloadindex' is set. > > Somebody should check. It would be interesting to hear about whether this > makes a performance impact, especially with slow filesystems and/or other > operating systems that have a relatively higher cost for 'lstat()'. > > Linus > > --- > builtin-checkout.c | 4 ++-- > diff-lib.c | 2 +- > 2 files changed, 3 insertions(+), 3 deletions(-) > > diff --git a/builtin-checkout.c b/builtin-checkout.c > index 15f0c32..3100ccd 100644 > --- a/builtin-checkout.c > +++ b/builtin-checkout.c > @@ -216,7 +216,7 @@ static int checkout_paths(struct tree *source_tree, const char **pathspec, > struct lock_file *lock_file = xcalloc(1, sizeof(struct lock_file)); > > newfd = hold_locked_index(lock_file, 1); > - if (read_cache() < 0) > + if (read_cache_preload(pathspec) < 0) > return error("corrupt index file"); > > if (source_tree) > @@ -367,7 +367,7 @@ static int merge_working_tree(struct checkout_opts *opts, > int newfd = hold_locked_index(lock_file, 1); > int reprime_cache_tree = 0; > > - if (read_cache() < 0) > + if (read_cache_preload(NULL) < 0) > return error("corrupt index file"); > > cache_tree_free(&active_cache_tree); > diff --git a/diff-lib.c b/diff-lib.c > index a310fb2..0aba6cd 100644 > --- a/diff-lib.c > +++ b/diff-lib.c > @@ -214,7 +214,7 @@ static int get_stat_data(struct cache_entry *ce, > const unsigned char *sha1 = ce->sha1; > unsigned int mode = ce->ce_mode; > > - if (!cached) { > + if (!cached && !ce_uptodate(ce)) { > int changed; > struct stat st; > changed = check_removed(ce, &st); > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 22:27 ` RE: david @ 2009-05-07 22:36 ` Linus Torvalds 2009-05-07 22:43 ` RE: david 0 siblings, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 22:36 UTC (permalink / raw) To: david; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, david@lang.hm wrote: > > his use case (as I understand it) is that the working tree is never updated by > anything other than git. it never recieves patches or manual edits. Well, you can certainly just use the CE_VALID bit in the index too (and this time I really mean CE_VALID). But it won't help anybody else, so it wouldn't be nearly as interesting. And I wonder how badly that code has rotted, thanks to not getting used. But yes, one thing to do would be git update-index --assume-unchanged --refresh which should hopefully set the bit, and then after that setting 'core.ignoreStat' should hopefully keep it set. Of course, you had then better _never_ make any mistakes and touch the files with non-git commands. And hope that the code still works ;) Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 22:36 ` RE: Linus Torvalds @ 2009-05-07 22:43 ` david 2009-05-07 23:00 ` RE: Linus Torvalds 0 siblings, 1 reply; 34+ messages in thread From: david @ 2009-05-07 22:43 UTC (permalink / raw) To: Linus Torvalds; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, Linus Torvalds wrote: > On Thu, 7 May 2009, david@lang.hm wrote: >> >> his use case (as I understand it) is that the working tree is never updated by >> anything other than git. it never recieves patches or manual edits. > > Well, you can certainly just use the CE_VALID bit in the index too (and > this time I really mean CE_VALID). But it won't help anybody else, so it > wouldn't be nearly as interesting. And I wonder how badly that code has > rotted, thanks to not getting used. > > But yes, one thing to do would be > > git update-index --assume-unchanged --refresh > > which should hopefully set the bit, and then after that setting > 'core.ignoreStat' should hopefully keep it set. > > Of course, you had then better _never_ make any mistakes and touch the > files with non-git commands. even with this a git checkout -f should replace all files, correct? David Lang > And hope that the code still works ;) > > Linus > ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 22:43 ` RE: david @ 2009-05-07 23:00 ` Linus Torvalds 2009-05-07 23:07 ` RE: david 0 siblings, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 23:00 UTC (permalink / raw) To: david; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, david@lang.hm wrote: > > even with this a git checkout -f should replace all files, correct? Hmm. I don't think so. As far as I recall, "-f" only overrides certain errors (like unmerged files or not up-to-date content), it doesn't change behavior wrt files that git thinks are already up-to-date. But I didn't check. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 23:00 ` RE: Linus Torvalds @ 2009-05-07 23:07 ` david 2009-05-07 23:18 ` RE: Linus Torvalds 0 siblings, 1 reply; 34+ messages in thread From: david @ 2009-05-07 23:07 UTC (permalink / raw) To: Linus Torvalds; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, Linus Torvalds wrote: > On Thu, 7 May 2009, david@lang.hm wrote: >> >> even with this a git checkout -f should replace all files, correct? > > Hmm. I don't think so. > > As far as I recall, "-f" only overrides certain errors (like unmerged > files or not up-to-date content), it doesn't change behavior wrt files > that git thinks are already up-to-date. what about a reset --hard? (is there any command that would force the files to be re-written, no matter what git thinks is already there) David Lang ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 23:07 ` RE: david @ 2009-05-07 23:18 ` Linus Torvalds 2009-05-07 23:31 ` RE: david 0 siblings, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-07 23:18 UTC (permalink / raw) To: david; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, david@lang.hm wrote: > > what about a reset --hard? (is there any command that would force the files to > be re-written, no matter what git thinks is already there) No, not "git reset --hard" either, I think. Git very much tries to avoid rewriting files, and if you've told it that file contents are stable, it will believe you. In fact, I think people used CE_VALID explicitly for the missing parts of "partial checkouts", so if we'd suddenly start writing files despite them being marked as ok in the tree, I think we'd have broken that part. (Although again - I'm not sure who would use CE_VALID and friends). If you want to force everything to be rewritten, you should just remove the index (or remove the specific entries in it if you want to do it just to a particular file) and then do a "git checkout" to re-read and re-populate the tree. But I'm not really seeing why you want to do this. If you told git that it shouldn't care about the working tree, why do you now want it do care? Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 23:18 ` RE: Linus Torvalds @ 2009-05-07 23:31 ` david 2009-05-07 23:57 ` Johan Herland 2009-05-08 16:14 ` Bevan Watkiss 0 siblings, 2 replies; 34+ messages in thread From: david @ 2009-05-07 23:31 UTC (permalink / raw) To: Linus Torvalds; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Thu, 7 May 2009, Linus Torvalds wrote: > On Thu, 7 May 2009, david@lang.hm wrote: >> >> what about a reset --hard? (is there any command that would force the files to >> be re-written, no matter what git thinks is already there) > > No, not "git reset --hard" either, I think. Git very much tries to avoid > rewriting files, and if you've told it that file contents are stable, it > will believe you. > > In fact, I think people used CE_VALID explicitly for the missing parts of > "partial checkouts", so if we'd suddenly start writing files despite them > being marked as ok in the tree, I think we'd have broken that part. > > (Although again - I'm not sure who would use CE_VALID and friends). > > If you want to force everything to be rewritten, you should just remove > the index (or remove the specific entries in it if you want to do it just > to a particular file) and then do a "git checkout" to re-read and > re-populate the tree. > > But I'm not really seeing why you want to do this. If you told git that it > shouldn't care about the working tree, why do you now want it do care? to be able to manually recover from the case where someone did things that they weren't supposed to removing the index and doing a checkout would be a reasonable thing to do (at least conceptually), I will admit that I don't remember ever seeing a command (or discussion of one) that would let me do that. ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-07 23:31 ` RE: david @ 2009-05-07 23:57 ` Johan Herland 2009-05-08 16:14 ` Bevan Watkiss 1 sibling, 0 replies; 34+ messages in thread From: Johan Herland @ 2009-05-07 23:57 UTC (permalink / raw) To: david; +Cc: git, Linus Torvalds, Bevan Watkiss, 'Alex Riesen' On Friday 08 May 2009, david@lang.hm wrote: > removing the index and doing a checkout would be a reasonable thing to do > (at least conceptually), I will admit that I don't remember ever seeing a > command (or discussion of one) that would let me do that. What about: rm .git/index git checkout -f or maybe: git update-index --no-assume-unchanged --refresh git checkout -f Hm? ....Johan -- Johan Herland, <johan@herland.net> www.herland.net ^ permalink raw reply [flat|nested] 34+ messages in thread
* RE: 2009-05-07 23:31 ` RE: david 2009-05-07 23:57 ` Johan Herland @ 2009-05-08 16:14 ` Bevan Watkiss 1 sibling, 0 replies; 34+ messages in thread From: Bevan Watkiss @ 2009-05-08 16:14 UTC (permalink / raw) To: david, 'Linus Torvalds' Cc: 'Alex Riesen', 'Git Mailing List' > -----Original Message----- > From: david@lang.hm [mailto:david@lang.hm] > Sent: May 7, 2009 7:31 PM > To: Linus Torvalds > Cc: Bevan Watkiss; 'Alex Riesen'; Git Mailing List > Subject: RE: > > On Thu, 7 May 2009, Linus Torvalds wrote: > > > On Thu, 7 May 2009, david@lang.hm wrote: > >> > >> what about a reset --hard? (is there any command that would force the > files to > >> be re-written, no matter what git thinks is already there) > > > > No, not "git reset --hard" either, I think. Git very much tries to avoid > > rewriting files, and if you've told it that file contents are stable, it > > will believe you. > > > > In fact, I think people used CE_VALID explicitly for the missing parts > of > > "partial checkouts", so if we'd suddenly start writing files despite > them > > being marked as ok in the tree, I think we'd have broken that part. > > > > (Although again - I'm not sure who would use CE_VALID and friends). > > > > If you want to force everything to be rewritten, you should just remove > > the index (or remove the specific entries in it if you want to do it > just > > to a particular file) and then do a "git checkout" to re-read and > > re-populate the tree. > > > > But I'm not really seeing why you want to do this. If you told git that > it > > shouldn't care about the working tree, why do you now want it do care? > > to be able to manually recover from the case where someone did things that > they weren't supposed to > > removing the index and doing a checkout would be a reasonable thing to do > (at least conceptually), I will admit that I don't remember ever seeing a > command (or discussion of one) that would let me do that. Added the patch and now the time is down to 4 1/2 minutes. Still a little slow for my needs though. Since I'm looking for a more instantaneous update I'll probably use something more along the lines of git fetch origin/master git log --name-only ..HEAD to get the list of files that have changed and copy them from a local repository. Nightly doing a real pull to confirm the files are correct and up to date. Bevan ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-07 21:55 ` Linus Torvalds 2009-05-07 22:27 ` RE: david @ 2009-05-08 8:17 ` Alex Riesen 2009-05-08 14:39 ` Re: Linus Torvalds 2009-05-08 16:47 ` 'git checkout' and unlink() calls (was: Re: ) Kjetil Barvik 2 siblings, 1 reply; 34+ messages in thread From: Alex Riesen @ 2009-05-08 8:17 UTC (permalink / raw) To: Linus Torvalds; +Cc: Bevan Watkiss, Git Mailing List 2009/5/7 Linus Torvalds <torvalds@linux-foundation.org>: > > Somebody should check. It would be interesting to hear about whether this > makes a performance impact, especially with slow filesystems and/or other > operating systems that have a relatively higher cost for 'lstat()'. > I did (cygwin). My guess, the improvement is completely dwarfed by the other overheads (like starting git and writing files). # Without the patch real 11m22.338s user 0m54.629s sys 8m33.638s # With checkout index preload real 11m14.361s user 0m46.609s sys 7m56.300s The script: #!/bin/sh if [ "$1" = setup ]; then for i in 1 2 3 4 do n=$(date) for f in `seq 1 10000` do echo "$n" >file$f done git add . printf "Commit $i:" git commit -m"$n" done exit fi export GIT_EXEC_PATH=/d/git-win time for f in `seq 1 10` do $GIT_EXEC_PATH/git checkout master~3 && $GIT_EXEC_PATH/git checkout master~2 && $GIT_EXEC_PATH/git checkout master~1 && $GIT_EXEC_PATH/git checkout master done exit ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 8:17 ` Alex Riesen @ 2009-05-08 14:39 ` Linus Torvalds 2009-05-08 15:51 ` Re: Brandon Casey 0 siblings, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-08 14:39 UTC (permalink / raw) To: Alex Riesen; +Cc: Bevan Watkiss, Git Mailing List On Fri, 8 May 2009, Alex Riesen wrote: > > I did (cygwin). My guess, the improvement is completely dwarfed by the > other overheads (like starting git and writing files). Oh, I meant "git checkout" as in not even switching branches, or perhaps switching branches but just changing a single file (among thousands). If you actually end up re-writing all files, then yes, it will obviously be totally dominated by other things. For example, in the kernel, switching between two branches that only differ in one file (Makefile) went from 0.18 seconds down to 0.14 seconds for me just because of the fewer lstat() calls. Noticeable? No. But it might be more noticeable on some other OS, or with some networked filesystem. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 14:39 ` Re: Linus Torvalds @ 2009-05-08 15:51 ` Brandon Casey 2009-05-08 16:15 ` Re: Linus Torvalds 0 siblings, 1 reply; 34+ messages in thread From: Brandon Casey @ 2009-05-08 15:51 UTC (permalink / raw) To: Linus Torvalds; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List Linus Torvalds wrote: > > On Fri, 8 May 2009, Alex Riesen wrote: >> I did (cygwin). My guess, the improvement is completely dwarfed by the >> other overheads (like starting git and writing files). > > Oh, I meant "git checkout" as in not even switching branches, or perhaps > switching branches but just changing a single file (among thousands). > > If you actually end up re-writing all files, then yes, it will obviously > be totally dominated by other things. > > For example, in the kernel, switching between two branches that only > differ in one file (Makefile) went from 0.18 seconds down to 0.14 seconds > for me just because of the fewer lstat() calls. > > Noticeable? No. But it might be more noticeable on some other OS, or with > some networked filesystem. plain 'git checkout' on linux kernel over NFS. Best time without patch: 1.20 seconds 0.45user 0.71system 0:01.20elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+15467minor)pagefaults 0swaps Best time with patch (core.preloadindex = true): 1.10 seconds 0.43user 4.00system 0:01.10elapsed 402%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+13999minor)pagefaults 0swaps Best time with patch (core.preloadindex = false): 0.84 seconds 0.42user 0.39system 0:00.84elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+13965minor)pagefaults 0swaps Best time with read_cache_preload patch only: 1.38 seconds 0.45user 4.42system 0:01.38elapsed 352%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+13990minor)pagefaults 0swaps The read_cache_preload() changes actually slow things down for me for this case. Reduction in lstat's gives a nice 30% improvement. -brandon ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 15:51 ` Re: Brandon Casey @ 2009-05-08 16:15 ` Linus Torvalds 2009-05-08 17:27 ` Re: Brandon Casey 0 siblings, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-08 16:15 UTC (permalink / raw) To: Brandon Casey; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List On Fri, 8 May 2009, Brandon Casey wrote: > > plain 'git checkout' on linux kernel over NFS. Thanks. > Best time without patch: 1.20 seconds > > 0.45user 0.71system 0:01.20elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+0outputs (0major+15467minor)pagefaults 0swaps > > Best time with patch (core.preloadindex = true): 1.10 seconds > > 0.43user 4.00system 0:01.10elapsed 402%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+0outputs (0major+13999minor)pagefaults 0swaps > > Best time with patch (core.preloadindex = false): 0.84 seconds > > 0.42user 0.39system 0:00.84elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+0outputs (0major+13965minor)pagefaults 0swaps Ok, that is _disgusting_. The parallelism clearly works (402%CPU), but the system time overhead is horrible. Going from 0.39s system time to 4s of system time is really quite nasty. Is there any possibility you could oprofile this (run it in a loop to get better profiles)? It very much sounds like some serious lock contention, and I'd love to hear more about exactly which lock it's hitting. Also, you're already almost totally CPU-bound, with 96% CPU for the single-threaded csase. So you may be running over NFS, but your NFS server is likely pretty good and/or the client just captures everything in the caches anyway. I don't recall what the Linux NFS stat cache timeout is, but it's less than a minute. I suspect that you ran things in a tight loop, which is why you then got effectively the local caching behavior for the best times. Can you do a "best time" check but with a 60-second pause between runs (and before), to see what happens when the client doesn't do caching? > Best time with read_cache_preload patch only: 1.38 seconds > > 0.45user 4.42system 0:01.38elapsed 352%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+0outputs (0major+13990minor)pagefaults 0swaps Yeah, here you're not getting any advantage of fewer lstats, and you show the same "almost entirely CPU-bound on four cores" behavior, and the same (probable) lock contention that has pushed the system time way up. > The read_cache_preload() changes actually slow things down for me for this > case. > > Reduction in lstat's gives a nice 30% improvement. Yes, I think the one-liner lstat avoidance is a real fix regardless. And the preloading sounds like it hits serialization overhead in the kernel, which I'm not at all surprised at, but not being surprised doesn't mean that I'm not interested to hear where it is. The Linux VFS dcache itself should scale better than that (but who knows - cacheline ping-pong due to lock contention can easily cause a 10x slowdown even without being _totally_ contended all the time). So I would _suspect_ that it's some NFS lock that you're seeing, but I'd love to know more. Btw, those system times are pretty high to begin with, so I'd love to know kernel version and see a profile even without the parallel case and presumably lock contention. Because while I probably have a faster machine anyway, what I see iis: [torvalds@nehalem linux]$ /usr/bin/time git checkout 0.13user 0.05system 0:00.19elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+13334minor)pagefaults 0swaps ie my "system" time is _much_ lower than yours (and lower than your system time). This is the 'without patch' time, btw, so this has extra lstat's. And my system time is still lower than my user time, so I wonder where all _your_ system time comes from. Your system time is much more comparable to user time even in the good case, and I wonder why? Could be just that kernel code tends to have more cache misses, and my 8MB cache captures it all, and yours doesn't. Regardless, a profile would be very interesting. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 16:15 ` Re: Linus Torvalds @ 2009-05-08 17:27 ` Brandon Casey 2009-05-08 17:43 ` Re: Brandon Casey 2009-05-08 17:44 ` Re: Linus Torvalds 0 siblings, 2 replies; 34+ messages in thread From: Brandon Casey @ 2009-05-08 17:27 UTC (permalink / raw) To: Linus Torvalds; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List Linus Torvalds wrote: > > On Fri, 8 May 2009, Brandon Casey wrote: >> plain 'git checkout' on linux kernel over NFS. > > Thanks. > >> Best time without patch: 1.20 seconds >> >> 0.45user 0.71system 0:01.20elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k >> 0inputs+0outputs (0major+15467minor)pagefaults 0swaps >> >> Best time with patch (core.preloadindex = true): 1.10 seconds >> >> 0.43user 4.00system 0:01.10elapsed 402%CPU (0avgtext+0avgdata 0maxresident)k >> 0inputs+0outputs (0major+13999minor)pagefaults 0swaps >> >> Best time with patch (core.preloadindex = false): 0.84 seconds >> >> 0.42user 0.39system 0:00.84elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k >> 0inputs+0outputs (0major+13965minor)pagefaults 0swaps > > Ok, that is _disgusting_. The parallelism clearly works (402%CPU), but the > system time overhead is horrible. Going from 0.39s system time to 4s of > system time is really quite nasty. > > Is there any possibility you could oprofile this (run it in a loop to get > better profiles)? It very much sounds like some serious lock contention, > and I'd love to hear more about exactly which lock it's hitting. Possibly, I'll see if our sysadmin has time to "play". > Also, you're already almost totally CPU-bound, with 96% CPU for the > single-threaded csase. So you may be running over NFS, but your NFS server > is likely pretty good and/or the client just captures everything in the > caches anyway. > > I don't recall what the Linux NFS stat cache timeout is, but it's less > than a minute. I suspect that you ran things in a tight loop, which is why > you then got effectively the local caching behavior for the best times. Yeah, that's what I did. > Can you do a "best time" check but with a 60-second pause between runs > (and before), to see what happens when the client doesn't do caching? No problem. >> Best time with read_cache_preload patch only: 1.38 seconds >> >> 0.45user 4.42system 0:01.38elapsed 352%CPU (0avgtext+0avgdata 0maxresident)k >> 0inputs+0outputs (0major+13990minor)pagefaults 0swaps > > Yeah, here you're not getting any advantage of fewer lstats, and you > show the same "almost entirely CPU-bound on four cores" behavior, and the > same (probable) lock contention that has pushed the system time way up. > >> The read_cache_preload() changes actually slow things down for me for this >> case. >> >> Reduction in lstat's gives a nice 30% improvement. > > Yes, I think the one-liner lstat avoidance is a real fix regardless. And > the preloading sounds like it hits serialization overhead in the kernel, > which I'm not at all surprised at, but not being surprised doesn't mean > that I'm not interested to hear where it is. > > The Linux VFS dcache itself should scale better than that (but who knows - > cacheline ping-pong due to lock contention can easily cause a 10x slowdown > even without being _totally_ contended all the time). So I would _suspect_ > that it's some NFS lock that you're seeing, but I'd love to know more. > > Btw, those system times are pretty high to begin with, so I'd love to know > kernel version and see a profile even without the parallel case and > presumably lock contention. Because while I probably have a faster > machine anyway, what I see iis: > > [torvalds@nehalem linux]$ /usr/bin/time git checkout > 0.13user 0.05system 0:00.19elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k > 0inputs+0outputs (0major+13334minor)pagefaults 0swaps > > ie my "system" time is _much_ lower than yours (and lower than your system > time). This is the 'without patch' time, btw, so this has extra lstat's. > And my system time is still lower than my user time, so I wonder where all > _your_ system time comes from. Your system time is much more comparable to > user time even in the good case, and I wonder why? > > Could be just that kernel code tends to have more cache misses, and my 8MB > cache captures it all, and yours doesn't. Regardless, a profile would be > very interesting. Something is definitely up. I provided timing results for your original preload_cache implementation which affected status and diff, which was part of the justification for merging it in. http://article.gmane.org/gmane.comp.version-control.git/100998 You can see that cold cache system time for 'git status' went from 0.36 to 0.52 seconds. Fine. I just ran it again, and now I'm getting system time of 10 seconds! This is the same machine. Similarly for the cold cache 'git checkout' reruns: Best without patch: 6.02 (systime 1.57) 0.43user 1.57system 0:06.02elapsed 33%CPU (0avgtext+0avgdata 0maxresident)k 5336inputs+0outputs (12major+15472minor)pagefaults 0swaps Best with patch (preload_cache,lstat reduction): 2.69 (systime 10.47) 0.45user 10.47system 0:02.69elapsed 405%CPU (0avgtext+0avgdata 0maxresident)k 5336inputs+0outputs (12major+13985minor)pagefaults 0swaps OS: Centos4.7 $ cat /proc/version Linux version 2.6.9-78.0.17.ELsmp (mockbuild@builder16.centos.org) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-9)) #1 SMP Thu Mar 12 20:05:15 EDT 2009 -brandon ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 17:27 ` Re: Brandon Casey @ 2009-05-08 17:43 ` Brandon Casey 2009-05-08 21:49 ` Re: Linus Torvalds 2009-05-08 17:44 ` Re: Linus Torvalds 1 sibling, 1 reply; 34+ messages in thread From: Brandon Casey @ 2009-05-08 17:43 UTC (permalink / raw) To: Linus Torvalds; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List Brandon Casey wrote: > Linus Torvalds wrote: >> And >> the preloading sounds like it hits serialization overhead in the kernel, >> which I'm not at all surprised at, but not being surprised doesn't mean >> that I'm not interested to hear where it is. >> >> The Linux VFS dcache itself should scale better than that (but who knows - >> cacheline ping-pong due to lock contention can easily cause a 10x slowdown >> even without being _totally_ contended all the time). So I would _suspect_ >> that it's some NFS lock that you're seeing, but I'd love to know more. >> >> Btw, those system times are pretty high to begin with, so I'd love to know >> kernel version and see a profile even without the parallel case and >> presumably lock contention. Here's an strace of 'git checkout': Before (cold cache): % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 98.60 6.365501 111 57432 lstat64 0.50 0.031984 359 89 2 close 0.25 0.015818 115 137 77 open 0.12 0.007670 23 339 write 0.09 0.005631 110 51 munmap 0.08 0.004873 49 99 69 stat64 0.07 0.004771 140 34 15 access 0.05 0.003083 280 11 5 waitpid 0.05 0.002973 10 284 brk 0.04 0.002816 469 6 execve <snip> After (cold cache, no lstat fix, just cache_preload): % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 90.90 23.717981 413 57432 lstat64 8.72 2.273917 162423 14 2 futex 0.12 0.032241 948 34 close 0.04 0.011507 202 57 munmap 0.04 0.009648 132 73 mmap2 0.03 0.008508 149 57 20 open 0.03 0.007771 311 25 mprotect 0.03 0.007758 388 20 clone 0.03 0.007548 23 334 write 0.02 0.005247 262 20 10 access -brandon ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 17:43 ` Re: Brandon Casey @ 2009-05-08 21:49 ` Linus Torvalds 2009-05-08 23:04 ` Re: Brandon Casey 0 siblings, 1 reply; 34+ messages in thread From: Linus Torvalds @ 2009-05-08 21:49 UTC (permalink / raw) To: Brandon Casey; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List On Fri, 8 May 2009, Brandon Casey wrote: > > Before (cold cache): > % time seconds usecs/call calls errors syscall > ------ ----------- ----------- --------- --------- ---------------- > 98.60 6.365501 111 57432 lstat64 > > After (cold cache, no lstat fix, just cache_preload): > % time seconds usecs/call calls errors syscall > ------ ----------- ----------- --------- --------- ---------------- > 90.90 23.717981 413 57432 lstat64 Yes, interesting. I really smells like it's all fixed performance and there is a single lock around it. That 111us -> 413us increase is very consistent with four cores all serializing on the same lock. So it parallelizes to all four cores, but then will take exactly as long in total. Quite frankly, 2.6.9 is so old that I have absolutely _no_ memory of what we used to do back then. Not that I follow NFS all that much even now - I did some of the original page cache and dentry work on the Linux NFS client way back when, but that was when I actually used NFS (and we were converting everything to the page cache). I've long since forgotten everything I knew, and I'm just as happy about that. But clearly something is bad, and equally clearly it worked much better for you a couple of months ago. Which does imply that there's probably some centos issues. Can you ask your MIS people if it would be possible to at least _test_ a new kernel? In 2.6.9, I'm quite frankly inclined to just say "it will likely never get fixed unless centos knows what it is", but if you test a more modern kernel and see similar issues, then I'll be intrigued. It's kind of sad, but at the same time, NFS was using the BKL up into 2.6.26 or something like that (about a year ago). And your kernel is based on something _much_ older. That said, even with the BKL, NFS should allow all the actual IO to be done in parallel (since the BKL is dropped on scheduling). But it's really wasting a _lot_ of CPU time, and that hurts you enormously, even though the cold-cache case still seems to win, judging by your other email: > Best without patch: 6.02 (systime 1.57) > > 0.43user 1.57system 0:06.02elapsed 33%CPU (0avgtext+0avgdata 0maxresident)k > 5336inputs+0outputs (12major+15472minor)pagefaults 0swaps > > Best with patch (preload_cache,lstat reduction): 2.69 (systime 10.47) > > 0.45user 10.47system 0:02.69elapsed 405%CPU (0avgtext+0avgdata 0maxresident)k > 5336inputs+0outputs (12major+13985minor)pagefaults 0swaps so there's a _huge_ increase in system time (again), but the change from 33% CPU -> 405% CPU makes up for it and you get lower elapsed times. But that 7x increase in system time really is sad. I do suspect it's likely due to spinning on the BKL. And if so, then a modern kernel should fix it. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 21:49 ` Re: Linus Torvalds @ 2009-05-08 23:04 ` Brandon Casey 2009-05-09 16:44 ` Re: Linus Torvalds 0 siblings, 1 reply; 34+ messages in thread From: Brandon Casey @ 2009-05-08 23:04 UTC (permalink / raw) To: Linus Torvalds; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List Linus Torvalds wrote: > > On Fri, 8 May 2009, Brandon Casey wrote: >> Before (cold cache): >> % time seconds usecs/call calls errors syscall >> ------ ----------- ----------- --------- --------- ---------------- >> 98.60 6.365501 111 57432 lstat64 >> >> After (cold cache, no lstat fix, just cache_preload): >> % time seconds usecs/call calls errors syscall >> ------ ----------- ----------- --------- --------- ---------------- >> 90.90 23.717981 413 57432 lstat64 > > Yes, interesting. I really smells like it's all fixed performance and > there is a single lock around it. That 111us -> 413us increase is very > consistent with four cores all serializing on the same lock. So it > parallelizes to all four cores, but then will take exactly as long in > total. Makes sense to me. > Quite frankly, 2.6.9 is so old that I have absolutely _no_ memory of what > we used to do back then. Not that I follow NFS all that much even now - I > did some of the original page cache and dentry work on the Linux NFS > client way back when, but that was when I actually used NFS (and we were > converting everything to the page cache). > > I've long since forgotten everything I knew, and I'm just as happy about > that. But clearly something is bad, and equally clearly it worked much > better for you a couple of months ago. Which does imply that there's > probably some centos issues. In case you're not aware CentOS is just repacked RHEL. I'm not sure if centos has the resources for investigating problems. We also have RHEL licenses, so hopefully I'll be able to come up with something to submit to them. > Can you ask your MIS people if it would be possible to at least _test_ a > new kernel? In 2.6.9, I'm quite frankly inclined to just say "it will > likely never get fixed unless centos knows what it is", but if you test a > more modern kernel and see similar issues, then I'll be intrigued. I think it's possible. Just not on this specific machine. Not sure what we have lying around multi-processor wise. Also, it won't happen until next week since it's late Friday afternoon here. btw, I've since done some more testing on some centos5.3 boxes we have. I get similar results (less ancient kernel 2.6.18). I've also scanned through the errata announcements that RedHat has released for their kernel updates. A few of them involve NFS. Possibly, whatever RedHat modified in the 5.X kernel was also backported to the 4.X kernel. > It's kind of sad, but at the same time, NFS was using the BKL up into > 2.6.26 or something like that (about a year ago). And your kernel is > based on something _much_ older. > > That said, even with the BKL, NFS should allow all the actual IO to be > done in parallel (since the BKL is dropped on scheduling). But it's really > wasting a _lot_ of CPU time, and that hurts you enormously, even though > the cold-cache case still seems to win, judging by your other email: > >> Best without patch: 6.02 (systime 1.57) >> >> 0.43user 1.57system 0:06.02elapsed 33%CPU (0avgtext+0avgdata 0maxresident)k >> 5336inputs+0outputs (12major+15472minor)pagefaults 0swaps >> >> Best with patch (preload_cache,lstat reduction): 2.69 (systime 10.47) >> >> 0.45user 10.47system 0:02.69elapsed 405%CPU (0avgtext+0avgdata 0maxresident)k >> 5336inputs+0outputs (12major+13985minor)pagefaults 0swaps > > so there's a _huge_ increase in system time (again), but the change from > 33% CPU -> 405% CPU makes up for it and you get lower elapsed times. > > But that 7x increase in system time really is sad. I do suspect it's > likely due to spinning on the BKL. And if so, then a modern kernel should > fix it. Thanks, I'll try to test next week. -brandon ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 23:04 ` Re: Brandon Casey @ 2009-05-09 16:44 ` Linus Torvalds 0 siblings, 0 replies; 34+ messages in thread From: Linus Torvalds @ 2009-05-09 16:44 UTC (permalink / raw) To: Brandon Casey; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List On Fri, 8 May 2009, Brandon Casey wrote: > > btw, I've since done some more testing on some centos5.3 boxes we have. > I get similar results (less ancient kernel 2.6.18). Yes, 2.6.18 is still much too old to matter from a locking standpoint. When people initially worried about scalability, the issues were more about server side stuff and the cached cases. NFS (as a client) is certainly used on the server side too, but it tends to be a somewhat secondary worry where only specific parts really matter. So people worked a lot more on the core kernel, and on local high-performance filesystem scaling. Only lately have we been pretty aggressive about finally really getting rid of the old "single big lock" (BKL) model entirely, or moving outwards from the core. And while we removed the BKL from the normal NFS read/write paths long long ago, all the name lookup and directory handling code still had it until a year ago. That, btw, is directly explained by perceived scalability issues: NFS is fairly often used as the backing store for a database and scaling thus matters there. But databases tend to keep their few big files open and use pread/pwrite - so pathname lookup is not nearly as significant for server ops as plain read/write. (Pathname lookup is important for things like web servers etc, but they rely heavily on caching for that, and the cached case scales fine). > I've also scanned through the errata announcements that RedHat has > released for their kernel updates. A few of them involve NFS. > Possibly, whatever RedHat modified in the 5.X kernel was also backported > to the 4.X kernel. That is very possibly the case. Expanding the BKL usage in some case could easily trigger the lock getting contention - and the way lock contention works, once you get a just even a small _hint_ of contention, things often fall off a cliff. The contention slows locking down, which in turn causes more CPU usage, which in turn causes _more_ contention. So even a small amount of extra locking - or even just slowing down some code that was inside the lock - can have catastrophic behavioural changes when the lock is close to being a problem. You do not get a nice gradual slowdown at all - you just hit a hard wall. I guess I should really try to set up some fileserver here at home to improve my test coverage. And to do better backups (or the little private data I have that I can't just mirror out to the world by turning it into an open-source project ;^) Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 2009-05-08 17:27 ` Re: Brandon Casey 2009-05-08 17:43 ` Re: Brandon Casey @ 2009-05-08 17:44 ` Linus Torvalds 1 sibling, 0 replies; 34+ messages in thread From: Linus Torvalds @ 2009-05-08 17:44 UTC (permalink / raw) To: Brandon Casey; +Cc: Alex Riesen, Bevan Watkiss, Git Mailing List On Fri, 8 May 2009, Brandon Casey wrote: > > Something is definitely up. > > I provided timing results for your original preload_cache implementation > which affected status and diff, which was part of the justification for > merging it in. > > http://article.gmane.org/gmane.comp.version-control.git/100998 > > You can see that cold cache system time for 'git status' went from 0.36 to > 0.52 seconds. Fine. I just ran it again, and now I'm getting system time > of 10 seconds! This is the same machine. Grr. > OS: Centos4.7 > > $ cat /proc/version > Linux version 2.6.9-78.0.17.ELsmp (mockbuild@builder16.centos.org) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-9)) #1 SMP Thu Mar 12 20:05:15 EDT 2009 Ok, if that's really the true kernel version (2.6.9), then that's some ancient kernel there. At the same time it's obviously been recompiled recently, so it got updated. At a guess, something got screwed up. But I have absolutely _no_ way to even guess what kernel patches centos puts in their ancient kernel builds. Perhaps a centos bugzilla entry might be appropriate? Somebody there might know what changed. Of course, it _could_ be an external change too, where the NFS server or timing changed just enough to trigger a pre-existing issue. But that would be pretty unlikely. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
* 'git checkout' and unlink() calls (was: Re: ) 2009-05-07 21:55 ` Linus Torvalds 2009-05-07 22:27 ` RE: david 2009-05-08 8:17 ` Alex Riesen @ 2009-05-08 16:47 ` Kjetil Barvik 2009-05-08 17:57 ` Linus Torvalds 2 siblings, 1 reply; 34+ messages in thread From: Kjetil Barvik @ 2009-05-08 16:47 UTC (permalink / raw) To: Linus Torvalds; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List * Linus Torvalds <torvalds@linux-foundation.org> writes: | So here's a better patch. It should cut down the 'lstat()' calls from | "git checkout" a lot. | | It looks obvious enough, and it passes testing (and now "git checkout" | only does about as many lstat's as there are files in the repository, | and they seem to all be properly asynchronous if 'core.preloadindex' | is set. I did a test by switching from v2.6.27 to v2.6.25, and now the only "lstat()-difference" between with and without the -q option is 2 lstat() calls extra done without the -q option. And, compared to over 41 000 lstat() calls, that is not noticable. Very good! | Somebody should check. It would be interesting to hear about whether | this makes a performance impact, especially with slow filesystems | and/or other operating systems that have a relatively higher cost for | 'lstat()'. Below is a table which is output from strace -o result -T git checkout my-v2.6.25 /* from my-v2.6.27 */ where the "result" file is run through a perl script to pretty print it: TOTAL 113988 100.000% OK:107252 NOT: 6736 6.263578 sec 55 usec/call lstat64 41114 36.069% OK: 35829 NOT: 5285 0.710936 sec 17 usec/call open 15027 13.183% OK: 13872 NOT: 1155 0.559302 sec 37 usec/call unlink 14379 12.614% OK: 14374 NOT: 5 3.720167 sec 259 usec/call write 14207 12.464% OK: 14207 NOT: 0 0.754196 sec 53 usec/call close 13872 12.170% OK: 13872 NOT: 0 0.185572 sec 13 usec/call fstat64 13862 12.161% OK: 13862 NOT: 0 0.169952 sec 12 usec/call rmdir 551 0.483% OK: 269 NOT: 282 0.035534 sec 64 usec/call brk 510 0.447% OK: 510 NOT: 0 0.014804 sec 29 usec/call mkdir 174 0.153% OK: 174 NOT: 0 0.102625 sec 590 usec/call mmap2 102 0.089% OK: 102 NOT: 0 0.001725 sec 17 usec/call read 68 0.060% OK: 68 NOT: 0 0.000999 sec 15 usec/call munmap 61 0.054% OK: 61 NOT: 0 0.005037 sec 83 usec/call access 20 0.018% OK: 12 NOT: 8 0.000348 sec 17 usec/call mprotect 13 0.011% OK: 13 NOT: 0 0.000193 sec 15 usec/call stat64 7 0.006% OK: 7 NOT: 0 0.000109 sec 16 usec/call getcwd 3 0.003% OK: 3 NOT: 0 0.000053 sec 18 usec/call chdir 3 0.003% OK: 3 NOT: 0 0.000048 sec 16 usec/call fcntl64 3 0.003% OK: 3 NOT: 0 0.000036 sec 12 usec/call rename 2 0.002% OK: 2 NOT: 0 0.001553 sec 776 usec/call setitimer 2 0.002% OK: 2 NOT: 0 0.000028 sec 14 usec/call getdents64 2 0.002% OK: 2 NOT: 0 0.000039 sec 20 usec/call uname 1 0.001% OK: 1 NOT: 0 0.000013 sec 13 usec/call time 1 0.001% OK: 1 NOT: 0 0.000011 sec 11 usec/call futex 1 0.001% OK: 1 NOT: 0 0.000013 sec 13 usec/call readlink 1 0.001% OK: 0 NOT: 1 0.000018 sec 18 usec/call execve 1 0.001% OK: 1 NOT: 0 0.000256 sec 256 usec/call getrlimit 1 0.001% OK: 1 NOT: 0 0.000011 sec 11 usec/call So, if the numbers from strace is trustable, 0.71 seconds is used on 41 114 calls to lstat64(). But, look at the unlink line, where each call took 259 microseconds (= 0.259 milliseconds), and all 14 379 calls took 3.72 seconds. It should be noted that when switching branch the other way (from .25 to .27), the unlink() calls used less time (below 160 microseconds each). Also note that the above was tested by only 3 runs. Warm cache. ext4 disk partition with git compiled with the USE_NSEC=1 option. Most (all?) of the unlink() calls seems to be from the following lines from the checkout_entry() funciton in entry.c /* * We unlink the old file, to get the new one with the * right permissions (including umask, which is nasty * to emulate by hand - much easier to let the system * just do the right thing) */ if (S_ISDIR(st.st_mode)) { /* If it is a gitlink, leave it alone! */ if (S_ISGITLINK(ce->ce_mode)) return 0; if (!state->force) return error("%s is a directory", path); remove_subtree(path); } else if (unlink(path)) return error("unable to unlink old '%s' (%s)", path, strerror(errno)); -- kjetil ^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: 'git checkout' and unlink() calls (was: Re: ) 2009-05-08 16:47 ` 'git checkout' and unlink() calls (was: Re: ) Kjetil Barvik @ 2009-05-08 17:57 ` Linus Torvalds 0 siblings, 0 replies; 34+ messages in thread From: Linus Torvalds @ 2009-05-08 17:57 UTC (permalink / raw) To: Kjetil Barvik; +Cc: Bevan Watkiss, 'Alex Riesen', Git Mailing List On Fri, 8 May 2009, Kjetil Barvik wrote: > > So, if the numbers from strace is trustable, 0.71 seconds is used on > 41 114 calls to lstat64(). But, look at the unlink line, where each > call took 259 microseconds (= 0.259 milliseconds), and all 14 379 > calls took 3.72 seconds. The system call times from strace are not really trustworthy. The overhead of tracing and in particular all the context switching back and forth between the tracer and the tracee means that the numbers should be taken with a large grain of salt. That said, they definitely aren't totally made up, and they tend to show real issues. In this particular case, what is going on is that 'lstat()' does no IO at all, while 'unlink()' generally at the very least will add things to some journal etc, and when the journal fills up, it will force IO. So doing 15k unlink() calls really is _much_ more expensive than doing 41k lstat() calls, since the latter will never force any IO at all (ok, so even doing just an lstat() may add atime updates etc to directories, but even if atime is enabled, that tends to only trigger one IO per second at most, and we never have to do any sync IO). > It should be noted that when switching branch the other way (from .25 > to .27), the unlink() calls used less time (below 160 microseconds > each). I don't think they are really "260 us each" or "160 us each". It's rather more likely that there are a few that are big due to forced IO, and most are in the couple of us case. Linus ^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2009-05-09 16:47 UTC | newest] Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-05-07 17:01 (unknown), Bevan Watkiss 2009-05-07 17:13 ` Alex Riesen 2009-05-07 17:26 ` Bevan Watkiss 2009-05-07 18:18 ` Alex Riesen 2009-05-07 18:48 ` Bevan Watkiss 2009-05-07 19:56 ` Björn Steinbrink 2009-05-07 18:56 ` Linus Torvalds 2009-05-07 19:37 ` RE: Bevan Watkiss 2009-05-07 20:07 ` RE: Linus Torvalds 2009-05-07 20:20 ` RE: Linus Torvalds 2009-05-07 20:43 ` Junio C Hamano 2009-05-07 21:33 ` Re: Linus Torvalds 2009-05-07 21:55 ` Linus Torvalds 2009-05-07 22:27 ` RE: david 2009-05-07 22:36 ` RE: Linus Torvalds 2009-05-07 22:43 ` RE: david 2009-05-07 23:00 ` RE: Linus Torvalds 2009-05-07 23:07 ` RE: david 2009-05-07 23:18 ` RE: Linus Torvalds 2009-05-07 23:31 ` RE: david 2009-05-07 23:57 ` Johan Herland 2009-05-08 16:14 ` Bevan Watkiss 2009-05-08 8:17 ` Alex Riesen 2009-05-08 14:39 ` Re: Linus Torvalds 2009-05-08 15:51 ` Re: Brandon Casey 2009-05-08 16:15 ` Re: Linus Torvalds 2009-05-08 17:27 ` Re: Brandon Casey 2009-05-08 17:43 ` Re: Brandon Casey 2009-05-08 21:49 ` Re: Linus Torvalds 2009-05-08 23:04 ` Re: Brandon Casey 2009-05-09 16:44 ` Re: Linus Torvalds 2009-05-08 17:44 ` Re: Linus Torvalds 2009-05-08 16:47 ` 'git checkout' and unlink() calls (was: Re: ) Kjetil Barvik 2009-05-08 17:57 ` Linus Torvalds
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).