Re: auto-packing on kernel.org? please?

[Carl Baldwin <cnb@fc.hp.com>]

I have a question about automatic repacking.

I am thinking of turning something like Linus' repacking heuristic loose
on my repositories.  I just want to make sure it is as safe as possible.

At the core of the incremental and full repack strategies are these
statements.

Incremental...
> 		git repack &&
> 			git prune-packed

Full...
> 		git repack -a -d &&
> 			git prune-packed

Are there some built in safety checks in 'git repack' and/or 'git
prune-packed' to guard against corruption?  In the long run, I would
feel more comfortable with somelike like this:

git repack
git verify-pack <new pack>
git prune-packed

Would something like this even work with 'git repack -a -d'?  Is there a
way to do something like the following for a full repack to achieve the
ultimate in paranoia?

git repack -a
git verify-pack <new pack file>
git trash-redundant-packs <new pack file>
git prune-packed

Carl

On Thu, Oct 13, 2005 at 11:44:30AM -0700, Linus Torvalds wrote:
> 
> I know we tried this once earlier, and it caused problems, but that was 
> when pack-files were new, and not everybody could handle them. These days, 
> if you can't handle pack-files, kernel.org is already pretty useless, 
> because all the major packages use them anyway, because people have 
> packed their repositories by hand.
> 
> So I'm suggesting we try to do an automatic repack every once in a while. 
> 
> In my suggestion, there would be two levels of repacking: "incremental" 
> and "full", and both of them would count the number of files before they 
> run, so that you'd only do it when it seems worthwhile.
> 
> This is a _really_ simple heuristic:
> 
>  - incremental repacking run every day:
> 
> 	#
> 	# Check if we have more than a couple of hundred
> 	# unpacked objects - approximated by whether we
> 	# have any "00" directory with more than one 
> 	#
> 	# This means that we don't repack projects that
> 	# that don't have a lot of work going on.
> 	#
> 	# Note: with really new versions of git, the "00"
> 	# directory may not exist if it has been pruned
> 	# away, so handle that gracefully.
> 	#
> 	export GIT_DIR=${1:-.}
> 	objs=$(find "$GIT_DIR/objects/00" -type f 2> /dev/null | wc -l)
> 	if [ "$obj" -gt 0 ]; then
> 		git repack &&
> 			git prune-packed
> 	fi
> 
>  - "full repack" every week if the number of packs has grown to be bigger 
>    than say 10 (ie even a very active projects will never have a full 
>    repack more than every other week)
> 
> 	#
> 	# Check if we have lots of packs, where "lots" is defined as 10.
> 	#
> 	# Note: with something that was generated with an old version
> 	# of git, the "pack" directory may not exist, so handle that
> 	# gracefully.
> 	#
> 	export GIT_DIR=${1:-.}
> 	packs=$(find "$GIT_DIR/objects/pack" -name '*.idx' 2> /dev/null | wc -l)
> 	if [ "$packs" -gt 10 ]; then
> 		git repack -a -d &&
> 			git prune-packed
> 	fi
> 
>  - do a full repack of everything once to start with.
> 
> 	export GIT_DIR=${1:-.}
> 	git repack -a -d &&
> 		git prune-packed
> 
> the above three trivial scripts just take a single argument, which becomes 
> the GIT_DIR (and if no argument exists, it would default to ".")
> 
> Is there any reason not to do this? Right now mirroring is slow, and 
> webgit is also getting to be very slow sometimes. I bet we'd be _much_ 
> better off with this kind of setup.
> 
> NOTE! The above is the "stupid" approach, which totally ignores alternate 
> directories, and isn't able to take advantage of the fact that many 
> projects could share objects. But it's simple, and it's efficient (eg it 
> won't spend time on things like the large historic archives which don't 
> change, but that would be expensive to repack if you didn't check for the 
> need).
> 
> So we could try to come up with a better approach eventually, which would 
> automatically notice alternate directories and not repack stuff that 
> exists there, but I'm pretty sure that the above would already help a 
> _lot_, and while pack-files have been been around forever, the 
> "alternates" support is still pretty new, so the above is also the "safer" 
> thing to do.
> 
> We'd only do the automatic thing on stuff under /pub/scm, of course: not 
> stuff in peoples home directories etc..
> 
> Peter?
> 
> 			Linus
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        Systems VLSI Laboratory
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -