Re: kernel.org mirroring (Re: [GIT PULL] MMC update)

From: Linus Torvalds <torvalds@osdl.org>
To: "H. Peter Anvin" <hpa@zytor.com>
Cc: Kernel Org Admin <ftpadmin@kernel.org>,
	Git Mailing List <git@vger.kernel.org>,
	Jakub Narebski <jnareb@gmail.com>
Subject: Re: kernel.org mirroring (Re: [GIT PULL] MMC update)
Date: Thu, 7 Dec 2006 11:05:59 -0800 (PST)	[thread overview]
Message-ID: <Pine.LNX.4.64.0612071052560.3615@woody.osdl.org> (raw)
In-Reply-To: <45785697.1060001@zytor.com>

On Thu, 7 Dec 2006, H. Peter Anvin wrote:
> 
> That all being said, the lack of intrinsic caching in gitweb continues to be a
> major problem for us.  Under high load, it makes all the problems worse.

I really don't see what gitweb could do that would be somehow better than 
apache doing the caching in front of it.. Is there some apache reason why 
that isn't sufficient (ie limitations on its cache size or timeouts?)

Maybe the cacheability hints from gitweb could be tweaked (a lot of it 
should be "infinitely cacheable", but the stuff that depends on refs and 
thus can change, could be set to some fixed host-wide value - preferably 
some that depends on how old the ref is).

Having gitweb be potentially up to an hour out of date is better than 
causing mirroring problems due to excessive load.

For example, if the git "refs/heads/" (or tags) directory hasn't changed 
in the last two months, we should probably set any ref-relative gitweb 
pages to have a caching timeout of a day or two. In contrast, if it's 
changed in the last hour, maybe we should only cache it for five minutes.

Jakub: any way to make gitweb set the "expires" fields _much_ more 
aggressively. I think we should at least have the ability to set a basic 
rules like

 - a _minimum_ of five minutes regardless of anything else

   We might even tweak this based on loadaverage, and it might be 
   worthwhile to add a randomization, to make sure that you don't get into 
   situations where everything webpage needs to be recalculated at once.

 - if refs/ directories are old, raise the minimum by the age of the refs

   If it's more than an hour old, raise it to ten minutes. If it's more 
   than a day, raise it to an hour. If it's more than a month old, raise 
   it to a day. And if it's more than half a year, it's some historical 
   archive like linux-history, and should probably default to a week or 
   more.

 - infinite for stuff that isn't ref-related.

Hmm?