gitweb.cgi

gitweb.cgi

[H. Peter Anvin]

It is increasingly clear that gitweb.cgi is producing an unacceptable 
load on the kernel.org servers.  Most of the hits we get are either the 
gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth 
like crazy.

This has become particularly painful during the current one-server outage.

Kay, gitweb really needs to be able to do caching, or be run behind a 
caching proxy.  Otherwise I will have to turn it off until we can come 
up with a dedicated piece of server hardware for it.

	-hpa

Re: gitweb.cgi

[Kay Sievers]

On Mon, Oct 17, 2005 at 07:57:22PM -0700, H. Peter Anvin wrote:
> It is increasingly clear that gitweb.cgi is producing an unacceptable 
> load on the kernel.org servers.

Sure, sorry, was on 3 conferences in a row the last weeks.

> Most of the hits we get are either the 
> gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth 
> like crazy.

I tested some stuff on these boxes and 30 stat() calls alone take app. 2 seconds
on these boxes cause of I/O load ... :)

> This has become particularly painful during the current one-server outage.
> 
> Kay, gitweb really needs to be able to do caching, or be run behind a 
> caching proxy.  Otherwise I will have to turn it off until we can come 
> up with a dedicated piece of server hardware for it.

How about Apache's mod_cache? Worked nicely for me several times in other
setups.

Kay

Re: gitweb.cgi

[H. Peter Anvin]

Kay Sievers wrote:
> 
>>This has become particularly painful during the current one-server outage.
>>
>>Kay, gitweb really needs to be able to do caching, or be run behind a 
>>caching proxy.  Otherwise I will have to turn it off until we can come 
>>up with a dedicated piece of server hardware for it.
> 
> How about Apache's mod_cache? Worked nicely for me several times in other
> setups.
> 

I will look at it and see if I can make it work properly.

	-hpa

Re: gitweb.cgi

[H. Peter Anvin]

Kay Sievers wrote:
> 
>>Most of the hits we get are either the 
>>gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth 
>>like crazy.
> 
> I tested some stuff on these boxes and 30 stat() calls alone take app. 2 seconds
> on these boxes cause of I/O load ... :)
> 

Welcome to my hell :)

I set up mod_cache (which I didn't know about, silly me) and so far it 
seems to work and has produced a tremendous decrease in load and 
improvement in response time.  I do, have, however, a request.  There 
are some gitweb pages which are more likely to change than others; in 
particular, some gitweb pages will *never* change (because they directly 
reflect immutable git data.)

If gitweb could produce Last-Modified and Expires headers where 
appropriate, it should improve caching performance.

	-hpa

Re: gitweb.cgi

[Brian Gerst]

H. Peter Anvin wrote:
> Kay Sievers wrote:
> 
>>
>>> Most of the hits we get are either the gitweb front page or the 
>>> gitweb rss feeds, and it's eating I/O bandwidth like crazy.
>>
>>
>> I tested some stuff on these boxes and 30 stat() calls alone take app. 
>> 2 seconds
>> on these boxes cause of I/O load ... :)
>>
> 
> Welcome to my hell :)
> 
> I set up mod_cache (which I didn't know about, silly me) and so far it 
> seems to work and has produced a tremendous decrease in load and 
> improvement in response time.  I do, have, however, a request.  There 
> are some gitweb pages which are more likely to change than others; in 
> particular, some gitweb pages will *never* change (because they directly 
> reflect immutable git data.)
> 
> If gitweb could produce Last-Modified and Expires headers where 
> appropriate, it should improve caching performance.
> 
>     -hpa

Some other areas for improvement would be to seperate out the git icon 
and the style sheet into seperate static files.

--
				Brian Gerst

Re: gitweb.cgi

[Linus Torvalds]

On Tue, 18 Oct 2005, H. Peter Anvin wrote:
> 
> I set up mod_cache (which I didn't know about, silly me) and so far it seems
> to work and has produced a tremendous decrease in load and improvement in
> response time.

It really doesn't work very well with the "front page" though..

Doing a "Save page as.." shows that it's not a huge page: it's roughly 700 
lines long and 57kB in size, but pressing the reload button (or just going 
somewhere else and coming back immediately) takes 45 seconds to reload for 
me.

Trying again shows that it _is_ cached if you press the reload button 
immediately again, but I haven't quite figured out how long the cache 
timeout is. It seems to be around one minute (from some very preliminary 
tests it's more than 25 seconds, but less than a minute and a half).

Considering that apparently the load is enough that it takes 45 seconds to 
generate (scary in itself), is should clearly be cached for more than one 
minute. More like ten minutes or half an hour, especially since mirroring 
any content changes takes longer than that anyway.

Now, I suspect all the content on kernel.org could easily be cached for 
ten minutes.

As far as I can tell, mod_cache without any expiry information uses

	CacheDefaultExpire 

which should default to one hour according to the docs. Have you changed 
that to one minute? Maybe making it 10 minutes would be better?

That said, I tried to figure out how the front page is generated, but 
haven't quite. Can somebody (Kay?) please say what it does most, and I can 
try to make sure git does that efficiently.. 

			Linus

Re: gitweb.cgi

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> It really doesn't work very well with the "front page" though..
> 
> Doing a "Save page as.." shows that it's not a huge page: it's roughly 700 
> lines long and 57kB in size, but pressing the reload button (or just going 
> somewhere else and coming back immediately) takes 45 seconds to reload for 
> me.
> 
> Trying again shows that it _is_ cached if you press the reload button 
> immediately again, but I haven't quite figured out how long the cache 
> timeout is. It seems to be around one minute (from some very preliminary 
> tests it's more than 25 seconds, but less than a minute and a half).
> 

The cache timeout is set to 300 seconds, however, that's per server, of 
course.

> Considering that apparently the load is enough that it takes 45 seconds to 
> generate (scary in itself), is should clearly be cached for more than one 
> minute. More like ten minutes or half an hour, especially since mirroring 
> any content changes takes longer than that anyway.

The latency for an I/O operation on the kernel.org servers is positively 
scary.

	-hpa

Re: gitweb.cgi

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> It really doesn't work very well with the "front page" though..
> 
> Doing a "Save page as.." shows that it's not a huge page: it's roughly 700 
> lines long and 57kB in size, but pressing the reload button (or just going 
> somewhere else and coming back immediately) takes 45 seconds to reload for 
> me.
> 
> Trying again shows that it _is_ cached if you press the reload button 
> immediately again, but I haven't quite figured out how long the cache 
> timeout is. It seems to be around one minute (from some very preliminary 
> tests it's more than 25 seconds, but less than a minute and a half).
> 

It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.

I also changed the CacheDefaultExpire to 600 seconds.

> That said, I tried to figure out how the front page is generated, but 
> haven't quite. Can somebody (Kay?) please say what it does most, and I can 
> try to make sure git does that efficiently.. 

The only thing the front page really should need is to know when the 
last change to the tree was, which presumably means looking at each head 
of each tree and follow the chain until there is a datable object.

	-hpa

Optimize common case of git-rev-list (was Re: gitweb.cgi)

[Linus Torvalds]

On Tue, 18 Oct 2005, H. Peter Anvin wrote:
> 
> > Considering that apparently the load is enough that it takes 45 seconds to
> > generate (scary in itself), is should clearly be cached for more than one
> > minute. More like ten minutes or half an hour, especially since mirroring
> > any content changes takes longer than that anyway.
> 
> The latency for an I/O operation on the kernel.org servers is positively
> scary.

I took a look at webgit, and it looks like at least for the "projects" 
page, the most common operation ends up being basically

	git-rev-list --header --parents --max-count=1 HEAD

Now, the thing is, the way "git-rev-list" works, it always keeps on 
popping the parents and parsing them in order to build the list of 
parents, and it turns out that even though we just want a single commit, 
git-rev-list will invariably look up _three_ generations of commits.

It will parse:
 - the commit we want (it obviously needs this)
 - it's parent(s) as part of the "pop_most_recent_commit()" logic
 - it will then pop one of the parents before it notices that it doesn't 
   need any more
 - and as part of popping the parent, it will parse the grandparent (again 
   due to "pop_most_recent_commit()".

Now, I've strace'd it, and it really is pretty efficient on the whole, but 
if things aren't nicely cached, and with long-latency IO, doing those two 
extra objects (at a minimum - if the parent is a merge it will be more) is 
just wasted time, and potentially a lot of it.

So here's a quick special-case for the trivial case of "just one commit, 
and no date-limits or other special rules".

Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---

I've actually tried to test it (but hey, "exhaustive" is hard with all the 
different options), and I tried to be very careful to only do the 
special-case when really nothing can go wrong, and this all looks obvious.

But buyer beware.

Btw, doing an "strace" on git-rev-list shows that most of the system calls 
by far are the dynamic loader. Now, those accesses _should_ all be cached, 
so they should be fast and low-latency, but it's entirely possible that 
for a server configuration you might want to actually link things 
statically. Or not. Just a thought.

diff --git a/rev-list.c b/rev-list.c
index c60aa72..d4da1bd 100644
--- a/rev-list.c
+++ b/rev-list.c
@@ -624,6 +624,10 @@ int main(int argc, char **argv)
 
 	if (!merge_order) {		
 		sort_by_date(&list);
+		if (list && !limited && max_count == 1) {
+			show_commit(list->item);
+			return 0;
+		}
 	        if (limited)
 			list = limit_list(list);
 		if (topo_order)

Re: gitweb.cgi

[Linus Torvalds]

On Tue, 18 Oct 2005, H. Peter Anvin wrote:
> 
> It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.
> 
> I also changed the CacheDefaultExpire to 600 seconds.

Ok, that sounds like it should improve things. My quick tests didn't seem 
to show any difference, though. Do you need to re-load the apache module 
or something?

> The only thing the front page really should need is to know when the last
> change to the tree was, which presumably means looking at each head of each
> tree and follow the chain until there is a datable object.

Yeah. I tried to follow gitweb.cgi, but I'm neither http- nor 
perl-literate, so I'm not sure I caught everything.

But it does seem to basically end up doing a "git_read_commit()" for each 
project, and that in turn was doing the "git-rev-list --max-count=1" thing 
that I just sent out a suggested improvement for.

It effectively removes two or more copies of

	stat64("/objects/xy/zzy", {...})
	fd = open("objects/xy/zzy", O_RDONLY|O_NOATIME)
	addr = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0)
	close(fd)
	munmap(addr, size)

which really should be very cheap operations, but hey, if the disk head is 
somewhere else (and busy) and it's not cached, it can be quite expensive. 
Especially since we don't end up usign the result.

I'm sure there's room for improvement inside gitweb itself too, but maybe 
the git-rev-list optimization will help.

		Linus

Re: gitweb.cgi

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> On Tue, 18 Oct 2005, H. Peter Anvin wrote:
> 
>>It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.
>>
>>I also changed the CacheDefaultExpire to 600 seconds.
> 
> 
> Ok, that sounds like it should improve things. My quick tests didn't seem 
> to show any difference, though. Do you need to re-load the apache module 
> or something?
> 

Yes, but I did that.  It seems very strange when something hits the 
cache.  A cgi script can apparently be run quite a few number of times 
before mod_cache sees it globally.

	-hpa

Re: gitweb.cgi

[Kay Sievers]

On Tue, Oct 18, 2005 at 06:02:29PM -0700, Linus Torvalds wrote:
> 
> 
> On Tue, 18 Oct 2005, H. Peter Anvin wrote:
> > 
> > It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.
> > 
> > I also changed the CacheDefaultExpire to 600 seconds.
> 
> Ok, that sounds like it should improve things. My quick tests didn't seem 
> to show any difference, though. Do you need to re-load the apache module 
> or something?
> 
> > The only thing the front page really should need is to know when the last
> > change to the tree was, which presumably means looking at each head of each
> > tree and follow the chain until there is a datable object.
> 
> Yeah. I tried to follow gitweb.cgi, but I'm neither http- nor 
> perl-literate, so I'm not sure I caught everything.
> 
> But it does seem to basically end up doing a "git_read_commit()" for each 
> project, and that in turn was doing the "git-rev-list --max-count=1" thing 
> that I just sent out a suggested improvement for.
> 
> It effectively removes two or more copies of
> 
> 	stat64("/objects/xy/zzy", {...})
> 	fd = open("objects/xy/zzy", O_RDONLY|O_NOATIME)
> 	addr = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0)
> 	close(fd)
> 	munmap(addr, size)
> 
> which really should be very cheap operations, but hey, if the disk head is 
> somewhere else (and busy) and it's not cached, it can be quite expensive. 
> Especially since we don't end up usign the result.
> 
> I'm sure there's room for improvement inside gitweb itself too, but maybe 
> the git-rev-list optimization will help.

There definitely is! But I tried a single "stat() all HEAD files" with a
simple script and it took more than 3 seconds for the 80 trees. Then I
gave up "optimizing" and was sure we want to have a single-file cached front
page instead. :)

Kay

Re: gitweb.cgi

[Kay Sievers]

On Tue, Oct 18, 2005 at 10:24:18AM -0700, H. Peter Anvin wrote:
> Kay Sievers wrote:
> >
> >>Most of the hits we get are either the 
> >>gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth 
> >>like crazy.
> >
> >I tested some stuff on these boxes and 30 stat() calls alone take app. 2 
> >seconds
> >on these boxes cause of I/O load ... :)
> >
> 
> Welcome to my hell :)

Yeah, I get an idea now :)

> I set up mod_cache (which I didn't know about, silly me) and so far it 
> seems to work and has produced a tremendous decrease in load and 
> improvement in response time.

Great! Hope that will work.

> I do, have, however, a request.  There 
> are some gitweb pages which are more likely to change than others; in 
> particular, some gitweb pages will *never* change (because they directly 
> reflect immutable git data.)

Yes, makes sense.

> If gitweb could produce Last-Modified and Expires headers where 
> appropriate, it should improve caching performance.

I've added the Expires: header to the commit and commitdiff pages with
one whole day ahead. Let's see if that will help...

Kay

Re: gitweb.cgi

[Linus Torvalds]

On Tue, 18 Oct 2005, H. Peter Anvin wrote:
> 
> Yes, but I did that.  It seems very strange when something hits the cache.  A
> cgi script can apparently be run quite a few number of times before mod_cache
> sees it globally.

Well, I tried again this morning, and _boy_ is it better. The "projects" 
thing came up immediately, no waiting. 

Maybe it just took a while for the mod_cache thing to take effect as 
Apache swarm members time out? 

Or maybe it's just that the west coast is really quiet at 7AM.

		Linus

Re: gitweb.cgi

[H. Peter Anvin]

Linus Torvalds wrote:
> 
> Well, I tried again this morning, and _boy_ is it better. The "projects" 
> thing came up immediately, no waiting. 
> 
> Maybe it just took a while for the mod_cache thing to take effect as 
> Apache swarm members time out? 
> 

No, that's not it.  It takes killing httpd and restarting it.  Rather, 
it looks like mod_cache in different swarm members doesn't always 
communicate instantly, which is also underscored by the fact that cache 
files don't appear into the filesystem until after a while.

	-hpa