* [PATCH 0/9] Gitweb caching v5
@ 2010-01-14 1:22 John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 1/9] gitweb: Load checking John 'Warthog9' Hawley
2010-01-15 1:40 ` [PATCH 0/9] Gitweb caching v5 Jakub Narebski
0 siblings, 2 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:22 UTC (permalink / raw)
To: git
Afternoon everyone,
This is the latest incarnation of gitweb w/ caching. This is
finally at the point where it should probably start either being
considered for inclusion or mainline, or I need to accept that this
will never get in and more perminantely fork (as is the case with
Fedora where this is going in as gitweb-caching as a parrallel rpm
package).
That said this brings the base up to mainline (again), it updates a
number of elements in the caching engine, and this is a much cleaner
break-out of the tree vs. what I am currently developing against.
v5:
- Missed a couple of things that were in my local tree, and
added them back in.
- Split up the die_error and the version matching patch
- Set version matching to be on by default - otherwise this
really is code that will never get checked, or at best
enabled by default by distributions
- Added a minor code cleanup with respect to $site_header
that was already in my tree
- Applied against a more recent git tree vs. 1.6.6-rc2
- Removed breakout patch for now (did that in v4 actually)
and will deal with that separately
http://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v5
v4:
- major re-working of the caching layer to use file handle
redirection instead of buffering output
- other minor improvements
http://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v4
v3:
- various minor re-works based on mailing list feedback,
this series was not sent to the mailing list.
v2:
- Better breakout
- You can actually disable the cache now
- John 'Warthog9' Hawley
John 'Warthog9' Hawley (9):
gitweb: Load checking
gitweb: change die_error to take "extra" argument for extended die
information
gitweb: Add option to force version match
gitweb: Makefile improvements
gitweb: add a get function to compliment print_local_time
gitweb: add a get function to compliment print_sort_th
gitweb: cleanup error message produced by undefined $site_header
gitweb: Convert output to using indirect file handle
gitweb: File based caching layer (from git.kernel.org)
Makefile | 65 +---
gitweb/Makefile | 129 +++++++
gitweb/README | 10 +-
gitweb/cache.pm | 283 ++++++++++++++
gitweb/gitweb.css | 6 +
gitweb/gitweb.perl | 1034 ++++++++++++++++++++++++++++++----------------------
6 files changed, 1030 insertions(+), 497 deletions(-)
create mode 100644 gitweb/Makefile
create mode 100644 gitweb/cache.pm
^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH 1/9] gitweb: Load checking
2010-01-14 1:22 [PATCH 0/9] Gitweb caching v5 John 'Warthog9' Hawley
@ 2010-01-14 1:22 ` John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information John 'Warthog9' Hawley
2010-01-15 22:30 ` [PATCH 1/9] gitweb: Load checking Jakub Narebski
2010-01-15 1:40 ` [PATCH 0/9] Gitweb caching v5 Jakub Narebski
1 sibling, 2 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:22 UTC (permalink / raw)
To: git
From: John 'Warthog9' Hawley <warthog9@kernel.org>
This changes slightly the behavior of gitweb, so that it verifies
that the box isn't inundated with before attempting to serve gitweb.
If the box is overloaded, it basically returns a 503 Server Unavailable
until the load falls below the defined threshold. This helps dramatically
if you have a box that's I/O bound, reaches a certain load and you
don't want gitweb, the I/O hog that it is, increasing the pain the
server is already undergoing.
This behavior is controlled by $maxload configuration variable.
Default is a load of 300, which for most cases should never be hit.
Unset it (set it to undefined value, i.e. undef) to turn off checking.
Currently it requires that '/proc/loadavg' file exists, otherwise the
load check is bypassed (load is taken to be 0). So platforms that do
not implement '/proc/loadavg' currently cannot use this feature.
(provisions are included for additional checks to be added by others)
Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
gitweb/README | 7 ++++++-
gitweb/gitweb.perl | 45 +++++++++++++++++++++++++++++++++++++++++----
2 files changed, 47 insertions(+), 5 deletions(-)
diff --git a/gitweb/README b/gitweb/README
index e34ee79..6c2c8e1 100644
--- a/gitweb/README
+++ b/gitweb/README
@@ -174,7 +174,7 @@ not include variables usually directly set during build):
Base URL for relative URLs in pages generated by gitweb,
(e.g. $logo, $favicon, @stylesheets if they are relative URLs),
needed and used only for URLs with nonempty PATH_INFO via
- <base href="$base_url>. Usually gitweb sets its value correctly,
+ <base href="$base_url">. Usually gitweb sets its value correctly,
and there is no need to set this variable, e.g. to $my_uri or "/".
* $home_link
Target of the home link on top of all pages (the first part of view
@@ -228,6 +228,11 @@ not include variables usually directly set during build):
repositories from launching cross-site scripting (XSS) attacks. Set this
to true if you don't trust the content of your repositories. The default
is false.
+ * $maxload
+ Used to set the maximum load that we will still respond to gitweb queries.
+ If server load exceed this value then return "503 Service Unavaliable" error.
+ Server load is taken to be 0 if gitweb cannot determine its value. Set it to
+ undefined value to turn it off. The default is 300.
Projects list file format
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 7e477af..0a07d3a 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -221,6 +221,12 @@ our %avatar_size = (
'double' => 32
);
+# Used to set the maximum load that we will still respond to gitweb queries.
+# If server load exceed this value then return "503 server busy" error.
+# If gitweb cannot determined server load, it is taken to be 0.
+# Leave it undefined (or set to 'undef') to turn off load checking.
+our $maxload = 300;
+
# You define site-wide feature defaults here; override them with
# $GITWEB_CONFIG as necessary.
our %feature = (
@@ -551,6 +557,32 @@ if (-e $GITWEB_CONFIG) {
do $GITWEB_CONFIG_SYSTEM if -e $GITWEB_CONFIG_SYSTEM;
}
+# Get loadavg of system, to compare against $maxload.
+# Currently it requires '/proc/loadavg' present to get loadavg;
+# if it is not present it returns 0, which means no load checking.
+sub get_loadavg {
+ if( -e '/proc/loadavg' ){
+ open my $fd, '<', '/proc/loadavg'
+ or return 0;
+ my @load = split(/\s+/, scalar <$fd>);
+ close $fd;
+
+ # The first three columns measure CPU and IO utilization of the last one,
+ # five, and 10 minute periods. The fourth column shows the number of
+ # currently running processes and the total number of processes in the m/n
+ # format. The last column displays the last process ID used.
+ return $load[0] || 0;
+ }
+ # additional checks for load average should go here for things that don't export
+ # /proc/loadavg
+
+ return 0;
+}
+
+if (defined $maxload && get_loadavg() > $maxload) {
+ die_error(503, "The load average on the server is too high");
+}
+
# version of the core git binary
our $git_version = qx("$GIT" --version) =~ m/git version (.*)$/ ? $1 : "unknown";
$number_of_git_cmds++;
@@ -3354,14 +3386,19 @@ sub git_footer_html {
# 500: The server isn't configured properly, or
# an internal error occurred (e.g. failed assertions caused by bugs), or
# an unknown error occurred (e.g. the git binary died unexpectedly).
+# 503: The server is currently unavailable (because it is overloaded,
+# or down for maintenance). Generally, this is a temporary state.
sub die_error {
my $status = shift || 500;
my $error = shift || "Internal server error";
- my %http_responses = (400 => '400 Bad Request',
- 403 => '403 Forbidden',
- 404 => '404 Not Found',
- 500 => '500 Internal Server Error');
+ my %http_responses = (
+ 400 => '400 Bad Request',
+ 403 => '403 Forbidden',
+ 404 => '404 Not Found',
+ 500 => '500 Internal Server Error',
+ 503 => '503 Service Unavailable',
+ );
git_header_html($http_responses{$status});
print <<EOF;
<div class="page_body">
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information
2010-01-14 1:22 ` [PATCH 1/9] gitweb: Load checking John 'Warthog9' Hawley
@ 2010-01-14 1:22 ` John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 3/9] gitweb: Add option to force version match John 'Warthog9' Hawley
2010-01-15 22:40 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information Jakub Narebski
2010-01-15 22:30 ` [PATCH 1/9] gitweb: Load checking Jakub Narebski
1 sibling, 2 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:22 UTC (permalink / raw)
To: git
This is a small change that just adds a 3rd, optional, parameter to die_error
that allows for extended error information to be output along with what the
error was.
---
gitweb/gitweb.perl | 8 +++++++-
1 files changed, 7 insertions(+), 1 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 0a07d3a..8298de5 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3391,6 +3391,7 @@ sub git_footer_html {
sub die_error {
my $status = shift || 500;
my $error = shift || "Internal server error";
+ my $extra = shift;
my %http_responses = (
400 => '400 Bad Request',
@@ -3405,8 +3406,13 @@ sub die_error {
<br /><br />
$status - $error
<br />
-</div>
EOF
+ if (defined $extra) {
+ print "<hr />\n" .
+ "$extra\n";
+ }
+ print "</div>\n";
+
git_footer_html();
exit;
}
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 3/9] gitweb: Add option to force version match
2010-01-14 1:22 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information John 'Warthog9' Hawley
@ 2010-01-14 1:22 ` John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 4/9] gitweb: Makefile improvements John 'Warthog9' Hawley
` (2 more replies)
2010-01-15 22:40 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information Jakub Narebski
1 sibling, 3 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:22 UTC (permalink / raw)
To: git
This adds $git_versions_must_match variable, which is set to true,
checks that we are running on the same version of git that we
shipped with, and if not throw '500 Internal Server Error' error.
What is checked is the version of gitweb (embedded in building
gitweb.cgi), against version of runtime git binary used.
Gitweb can usually run with a mismatched git install. This is more
here to give an obvious warning as to whats going on vs. silently
failing.
By default this feature is turned on.
Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
gitweb/README | 3 +++
gitweb/gitweb.perl | 23 +++++++++++++++++++++++
2 files changed, 26 insertions(+), 0 deletions(-)
diff --git a/gitweb/README b/gitweb/README
index 6c2c8e1..03151d2 100644
--- a/gitweb/README
+++ b/gitweb/README
@@ -233,6 +233,9 @@ not include variables usually directly set during build):
If server load exceed this value then return "503 Service Unavaliable" error.
Server load is taken to be 0 if gitweb cannot determine its value. Set it to
undefined value to turn it off. The default is 300.
+ * $git_versions_must_match
+ If set, gitweb fails with 500 Internal Server Error if the version of gitweb
+ doesn't match version of git binary. The default is true.
Projects list file format
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 8298de5..b41bc33 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -221,6 +221,9 @@ our %avatar_size = (
'double' => 32
);
+# If it is true, exit if gitweb version and git binary version don't match
+our $git_versions_must_match = 1;
+
# Used to set the maximum load that we will still respond to gitweb queries.
# If server load exceed this value then return "503 server busy" error.
# If gitweb cannot determined server load, it is taken to be 0.
@@ -587,6 +590,26 @@ if (defined $maxload && get_loadavg() > $maxload) {
our $git_version = qx("$GIT" --version) =~ m/git version (.*)$/ ? $1 : "unknown";
$number_of_git_cmds++;
+# Throw an error if git versions does not match, if $git_versions_must_match is true.
+if ($git_versions_must_match &&
+ $git_version ne $version) {
+ my $admin_contact =
+ defined $ENV{'SERVER_ADMIN'} ? ", $ENV{'SERVER_ADMIN'}," : '';
+ my $err_msg = <<EOT;
+<h1 align="center">*** Warning ***</h1>
+<p>
+This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,
+however git version <b>@{[esc_html($git_version)]}</b> was found on server,
+and administrator requested strict version checking.
+</p>
+<p>
+Please contact the server administrator${admin_contact} to either configure
+gitweb to allow mismatched versions, or update git or gitweb installation.
+</p>
+EOT
+ die_error(500, 'Internal server error', $err_msg);
+}
+
$projects_list ||= $projectroot;
# ======================================================================
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 4/9] gitweb: Makefile improvements
2010-01-14 1:22 ` [PATCH 3/9] gitweb: Add option to force version match John 'Warthog9' Hawley
@ 2010-01-14 1:23 ` John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 5/9] gitweb: add a get function to compliment print_local_time John 'Warthog9' Hawley
2010-01-15 23:36 ` [PATCH 3/9] gitweb: Add option to force version match Jakub Narebski
2010-01-24 21:59 ` Petr Baudis
2 siblings, 1 reply; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:23 UTC (permalink / raw)
To: git
From: John 'Warthog9' Hawley <warthog9@kernel.org>
This commit adjust the main Makefile so you can simply run
make gitweb
which in turn calls gitweb/Makefile. This means that in order to
generate gitweb, you can simply run 'make' from gitweb subdirectory:
cd gitweb
make
Targets gitweb/gitweb.cgi and (dependent on JSMIN being defined)
gitweb/gitweb.min.js in main Makefile are preserved for backward
compatibility.
Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
Makefile | 65 +++++----------------------
gitweb/Makefile | 129 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 141 insertions(+), 53 deletions(-)
create mode 100644 gitweb/Makefile
diff --git a/Makefile b/Makefile
index be18389..9f069a8 100644
--- a/Makefile
+++ b/Makefile
@@ -282,29 +282,6 @@ pathsep = :
# JavaScript minifier invocation that can function as filter
JSMIN =
-# default configuration for gitweb
-GITWEB_CONFIG = gitweb_config.perl
-GITWEB_CONFIG_SYSTEM = /etc/gitweb.conf
-GITWEB_HOME_LINK_STR = projects
-GITWEB_SITENAME =
-GITWEB_PROJECTROOT = /pub/git
-GITWEB_PROJECT_MAXDEPTH = 2007
-GITWEB_EXPORT_OK =
-GITWEB_STRICT_EXPORT =
-GITWEB_BASE_URL =
-GITWEB_LIST =
-GITWEB_HOMETEXT = indextext.html
-GITWEB_CSS = gitweb.css
-GITWEB_LOGO = git-logo.png
-GITWEB_FAVICON = git-favicon.png
-ifdef JSMIN
-GITWEB_JS = gitweb.min.js
-else
-GITWEB_JS = gitweb.js
-endif
-GITWEB_SITE_HEADER =
-GITWEB_SITE_FOOTER =
-
export prefix bindir sharedir sysconfdir
CC = gcc
@@ -1535,6 +1512,11 @@ $(patsubst %.perl,%,$(SCRIPT_PERL)): % : %.perl
chmod +x $@+ && \
mv $@+ $@
+
+.PHONY: gitweb
+gitweb:
+ $(QUIET_SUBDIR0)gitweb $(QUIET_SUBDIR1) all
+
ifdef JSMIN
OTHER_PROGRAMS += gitweb/gitweb.cgi gitweb/gitweb.min.js
gitweb/gitweb.cgi: gitweb/gitweb.perl gitweb/gitweb.min.js
@@ -1542,30 +1524,13 @@ else
OTHER_PROGRAMS += gitweb/gitweb.cgi
gitweb/gitweb.cgi: gitweb/gitweb.perl
endif
- $(QUIET_GEN)$(RM) $@ $@+ && \
- sed -e '1s|#!.*perl|#!$(PERL_PATH_SQ)|' \
- -e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \
- -e 's|++GIT_BINDIR++|$(bindir)|g' \
- -e 's|++GITWEB_CONFIG++|$(GITWEB_CONFIG)|g' \
- -e 's|++GITWEB_CONFIG_SYSTEM++|$(GITWEB_CONFIG_SYSTEM)|g' \
- -e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \
- -e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \
- -e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \
- -e 's|"++GITWEB_PROJECT_MAXDEPTH++"|$(GITWEB_PROJECT_MAXDEPTH)|g' \
- -e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \
- -e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \
- -e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \
- -e 's|++GITWEB_LIST++|$(GITWEB_LIST)|g' \
- -e 's|++GITWEB_HOMETEXT++|$(GITWEB_HOMETEXT)|g' \
- -e 's|++GITWEB_CSS++|$(GITWEB_CSS)|g' \
- -e 's|++GITWEB_LOGO++|$(GITWEB_LOGO)|g' \
- -e 's|++GITWEB_FAVICON++|$(GITWEB_FAVICON)|g' \
- -e 's|++GITWEB_JS++|$(GITWEB_JS)|g' \
- -e 's|++GITWEB_SITE_HEADER++|$(GITWEB_SITE_HEADER)|g' \
- -e 's|++GITWEB_SITE_FOOTER++|$(GITWEB_SITE_FOOTER)|g' \
- $< >$@+ && \
- chmod +x $@+ && \
- mv $@+ $@
+ $(QUIET_SUBDIR0)gitweb $(QUIET_SUBDIR1) $(patsubst gitweb/%,%,$@)
+
+ifdef JSMIN
+gitweb/gitweb.min.js: gitweb/gitweb.js
+ $(QUIET_SUBDIR0)gitweb $(QUIET_SUBDIR1) $(patsubst gitweb/%,%,$@)
+endif # JSMIN
+
git-instaweb: git-instaweb.sh gitweb/gitweb.cgi gitweb/gitweb.css gitweb/gitweb.js
$(QUIET_GEN)$(RM) $@ $@+ && \
@@ -1592,12 +1557,6 @@ $(patsubst %.perl,%,$(SCRIPT_PERL)) git-instaweb: % : unimplemented.sh
mv $@+ $@
endif # NO_PERL
-
-ifdef JSMIN
-gitweb/gitweb.min.js: gitweb/gitweb.js
- $(QUIET_GEN)$(JSMIN) <$< >$@
-endif # JSMIN
-
ifndef NO_PYTHON
$(patsubst %.py,%,$(SCRIPT_PYTHON)): GIT-CFLAGS
$(patsubst %.py,%,$(SCRIPT_PYTHON)): % : %.py
diff --git a/gitweb/Makefile b/gitweb/Makefile
new file mode 100644
index 0000000..c9eb1ee
--- /dev/null
+++ b/gitweb/Makefile
@@ -0,0 +1,129 @@
+# The default target of this Makefile is...
+all::
+
+# Define V=1 to have a more verbose compile.
+#
+# Define JSMIN to point to JavaScript minifier that functions as
+# a filter to have gitweb.js minified.
+#
+
+prefix ?= $(HOME)
+bindir ?= $(prefix)/bin
+RM ?= rm -f
+
+# JavaScript minifier invocation that can function as filter
+JSMIN ?=
+
+# default configuration for gitweb
+GITWEB_CONFIG = gitweb_config.perl
+GITWEB_CONFIG_SYSTEM = /etc/gitweb.conf
+GITWEB_HOME_LINK_STR = projects
+GITWEB_SITENAME =
+GITWEB_PROJECTROOT = /pub/git
+GITWEB_PROJECT_MAXDEPTH = 2007
+GITWEB_EXPORT_OK =
+GITWEB_STRICT_EXPORT =
+GITWEB_BASE_URL =
+GITWEB_LIST =
+GITWEB_HOMETEXT = indextext.html
+GITWEB_CSS = gitweb.css
+GITWEB_LOGO = git-logo.png
+GITWEB_FAVICON = git-favicon.png
+ifdef JSMIN
+GITWEB_JS = gitweb.min.js
+else
+GITWEB_JS = gitweb.js
+endif
+GITWEB_SITE_HEADER =
+GITWEB_SITE_FOOTER =
+
+# include user config
+-include ../config.mak.autogen
+-include ../config.mak
+
+# determine version
+../GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE
+ $(QUIET_SUBDIR0)../ $(QUIET_SUBDIR1) GIT-VERSION-FILE
+
+-include ../GIT-VERSION-FILE
+
+### Build rules
+
+SHELL_PATH ?= $(SHELL)
+PERL_PATH ?= /usr/bin/perl
+
+# Shell quote;
+bindir_SQ = $(subst ','\'',$(bindir)) #'
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH)) #'
+PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH)) #'
+
+# Quiet generation (unless V=1)
+QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1 =
+
+ifneq ($(findstring $(MAKEFLAGS),w),w)
+PRINT_DIR = --no-print-directory
+else # "make -w"
+NO_SUBDIR = :
+endif
+
+ifneq ($(findstring $(MAKEFLAGS),s),s)
+ifndef V
+ QUIET = @
+ QUIET_GEN = $(QUIET)echo ' ' GEN $@;
+ QUIET_SUBDIR0 = +@subdir=
+ QUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \
+ $(MAKE) $(PRINT_DIR) -C $$subdir
+ export V
+ export QUIET
+ export QUIET_GEN
+ export QUIET_SUBDIR0
+ export QUIET_SUBDIR1
+endif
+endif
+
+all:: gitweb.cgi
+
+ifdef JSMIN
+FILES=gitweb.cgi gitweb.min.js
+gitweb.cgi: gitweb.perl gitweb.min.js
+else # !JSMIN
+FILES=gitweb.cgi
+gitweb.cgi: gitweb.perl
+endif # JSMIN
+
+gitweb.cgi:
+ $(QUIET_GEN)$(RM) $@ $@+ && \
+ sed -e '1s|#!.*perl|#!$(PERL_PATH_SQ)|' \
+ -e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \
+ -e 's|++GIT_BINDIR++|$(bindir)|g' \
+ -e 's|++GITWEB_CONFIG++|$(GITWEB_CONFIG)|g' \
+ -e 's|++GITWEB_CONFIG_SYSTEM++|$(GITWEB_CONFIG_SYSTEM)|g' \
+ -e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \
+ -e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \
+ -e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \
+ -e 's|"++GITWEB_PROJECT_MAXDEPTH++"|$(GITWEB_PROJECT_MAXDEPTH)|g' \
+ -e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \
+ -e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \
+ -e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \
+ -e 's|++GITWEB_LIST++|$(GITWEB_LIST)|g' \
+ -e 's|++GITWEB_HOMETEXT++|$(GITWEB_HOMETEXT)|g' \
+ -e 's|++GITWEB_CSS++|$(GITWEB_CSS)|g' \
+ -e 's|++GITWEB_LOGO++|$(GITWEB_LOGO)|g' \
+ -e 's|++GITWEB_FAVICON++|$(GITWEB_FAVICON)|g' \
+ -e 's|++GITWEB_JS++|$(GITWEB_JS)|g' \
+ -e 's|++GITWEB_SITE_HEADER++|$(GITWEB_SITE_HEADER)|g' \
+ -e 's|++GITWEB_SITE_FOOTER++|$(GITWEB_SITE_FOOTER)|g' \
+ $< >$@+ && \
+ chmod +x $@+ && \
+ mv $@+ $@
+
+ifdef JSMIN
+gitweb.min.js: gitweb.js
+ $(QUIET_GEN)$(JSMIN) <$< >$@
+endif # JSMIN
+
+clean:
+ $(RM) $(FILES)
+
+.PHONY: all clean .FORCE-GIT-VERSION-FILE
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 5/9] gitweb: add a get function to compliment print_local_time
2010-01-14 1:23 ` [PATCH 4/9] gitweb: Makefile improvements John 'Warthog9' Hawley
@ 2010-01-14 1:23 ` John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 6/9] gitweb: add a get function to compliment print_sort_th John 'Warthog9' Hawley
2010-01-23 11:13 ` [PATCH 5/9] gitweb: add a get function to compliment print_local_time Jakub Narebski
0 siblings, 2 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:23 UTC (permalink / raw)
To: git
This adds a get function for print_local_time so that the basic
function can be used outside of their straight printing operation.
---
gitweb/gitweb.perl | 11 +++++++++--
1 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index b41bc33..d38aad6 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3537,14 +3537,21 @@ sub git_print_header_div {
}
sub print_local_time {
+ print get_local_time(@_);
+}
+
+sub get_local_time {
+ my $localtime = "";
my %date = @_;
if ($date{'hour_local'} < 6) {
- printf(" (<span class=\"atnight\">%02d:%02d</span> %s)",
+ $localtime .= sprintf(" (<span class=\"atnight\">%02d:%02d</span> %s)",
$date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});
} else {
- printf(" (%02d:%02d %s)",
+ $localtime .= sprintf(" (%02d:%02d %s)",
$date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});
}
+
+ return $localtime
}
# Outputs the author name and date in long form
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 6/9] gitweb: add a get function to compliment print_sort_th
2010-01-14 1:23 ` [PATCH 5/9] gitweb: add a get function to compliment print_local_time John 'Warthog9' Hawley
@ 2010-01-14 1:23 ` John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header John 'Warthog9' Hawley
2010-01-23 11:13 ` [PATCH 5/9] gitweb: add a get function to compliment print_local_time Jakub Narebski
1 sibling, 1 reply; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:23 UTC (permalink / raw)
To: git
This adds a get function for print_sort_th so that the basic
function can be used outside of their straight printing operation.
---
gitweb/gitweb.perl | 11 +++++++++--
1 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index d38aad6..07fdeb5 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -4375,17 +4375,24 @@ sub fill_project_list_info {
# print 'sort by' <th> element, generating 'sort by $name' replay link
# if that order is not selected
sub print_sort_th {
+ print get_sort_th(@_);
+}
+
+sub get_sort_th {
my ($name, $order, $header) = @_;
+ my $sortth = "";
$header ||= ucfirst($name);
if ($order eq $name) {
- print "<th>$header</th>\n";
+ $sortth .= "<th>$header</th>\n";
} else {
- print "<th>" .
+ $sortth .= "<th>" .
$cgi->a({-href => href(-replay=>1, order=>$name),
-class => "header"}, $header) .
"</th>\n";
}
+
+ return $sortth;
}
sub git_project_list_body {
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header
2010-01-14 1:23 ` [PATCH 6/9] gitweb: add a get function to compliment print_sort_th John 'Warthog9' Hawley
@ 2010-01-14 1:23 ` John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle John 'Warthog9' Hawley
2010-01-15 23:49 ` [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header Jakub Narebski
0 siblings, 2 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:23 UTC (permalink / raw)
To: git
If $site_header is not defined you get extraneous errors in the web
logs:
[Wed Jan 13 16:55:42 2010] [error] [client ::1] [Wed Jan 13 16:55:42 2010] gitweb.cgi: Use of uninitialized value $site_header in -f at /var/www/gitweb/gitweb.cgi line 3287., referer: http://git/gitweb.cgi
for example. This ensures that the variable is defined before trying to use it.
---
gitweb/gitweb.perl | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 07fdeb5..c4a177d 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3277,7 +3277,7 @@ EOF
print "</head>\n" .
"<body>\n";
- if (-f $site_header) {
+ if ($site_header && -f $site_header) {
insert_file($site_header);
}
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-14 1:23 ` [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header John 'Warthog9' Hawley
@ 2010-01-14 1:23 ` John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org) John 'Warthog9' Hawley
2010-01-16 0:43 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle Jakub Narebski
2010-01-15 23:49 ` [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header Jakub Narebski
1 sibling, 2 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:23 UTC (permalink / raw)
To: git
This converts the output handling of gitweb to using an indirect
file handle. This is in preparation to add the caching layer. This
is a slight modification to the way I was originally doing it by
passing the output around. This should be a nop and this shouldn't
change the behavior of gitweb. This does leave error reporting
functions (die_error specifically) continuing to output directly
as I want to garauntee those will report their errors regardless of
what may be going on with respect to the rest of the output.
---
gitweb/gitweb.perl | 880 ++++++++++++++++++++++++++--------------------------
1 files changed, 448 insertions(+), 432 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index c4a177d..8bb323c 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -450,6 +450,13 @@ our %feature = (
'default' => [0]},
);
+# Basic file handler for all of gitweb, there are two of them. The first
+# is the basic text/html file handler which is used for everything other
+# then the binary files, that uses a separate file handler though
+# these are both set to STDOUT for the time being.
+our $output_handler = *STDOUT;
+our $output_handler_bin = *STDOUT;
+
sub gitweb_get_feature {
my ($name) = @_;
return unless exists $feature{$name};
@@ -3081,7 +3088,7 @@ sub insert_file {
my $filename = shift;
open my $fd, '<', $filename;
- print map { to_utf8($_) } <$fd>;
+ print {$output_handler} map { to_utf8($_) } <$fd>;
close $fd;
}
@@ -3198,10 +3205,10 @@ sub git_header_html {
} else {
$content_type = 'text/html';
}
- print $cgi->header(-type=>$content_type, -charset => 'utf-8',
+ print {$output_handler} $cgi->header(-type=>$content_type, -charset => 'utf-8',
-status=> $status, -expires => $expires);
my $mod_perl_version = $ENV{'MOD_PERL'} ? " $ENV{'MOD_PERL'}" : '';
- print <<EOF;
+ print {$output_handler} <<EOF;
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-US" lang="en-US">
@@ -3216,16 +3223,16 @@ EOF
# the stylesheet, favicon etc urls won't work correctly with path_info
# unless we set the appropriate base URL
if ($ENV{'PATH_INFO'}) {
- print "<base href=\"".esc_url($base_url)."\" />\n";
+ print {$output_handler} "<base href=\"".esc_url($base_url)."\" />\n";
}
# print out each stylesheet that exist, providing backwards capability
# for those people who defined $stylesheet in a config file
if (defined $stylesheet) {
- print '<link rel="stylesheet" type="text/css" href="'.$stylesheet.'"/>'."\n";
+ print {$output_handler} '<link rel="stylesheet" type="text/css" href="'.$stylesheet.'"/>'."\n";
} else {
foreach my $stylesheet (@stylesheets) {
next unless $stylesheet;
- print '<link rel="stylesheet" type="text/css" href="'.$stylesheet.'"/>'."\n";
+ print {$output_handler} '<link rel="stylesheet" type="text/css" href="'.$stylesheet.'"/>'."\n";
}
}
if (defined $project) {
@@ -3244,7 +3251,7 @@ EOF
$href_params{'action'} = $type;
$link_attr{'-href'} = href(%href_params);
- print "<link ".
+ print {$output_handler} "<link ".
"rel=\"$link_attr{'-rel'}\" ".
"title=\"$link_attr{'-title'}\" ".
"href=\"$link_attr{'-href'}\" ".
@@ -3254,7 +3261,7 @@ EOF
$href_params{'extra_options'} = '--no-merges';
$link_attr{'-href'} = href(%href_params);
$link_attr{'-title'} .= ' (no merges)';
- print "<link ".
+ print {$output_handler} "<link ".
"rel=\"$link_attr{'-rel'}\" ".
"title=\"$link_attr{'-title'}\" ".
"href=\"$link_attr{'-href'}\" ".
@@ -3263,37 +3270,37 @@ EOF
}
} else {
- printf('<link rel="alternate" title="%s projects list" '.
+ printf( {$output_handler} '<link rel="alternate" title="%s projects list" '.
'href="%s" type="text/plain; charset=utf-8" />'."\n",
$site_name, href(project=>undef, action=>"project_index"));
- printf('<link rel="alternate" title="%s projects feeds" '.
+ printf( {$output_handler} '<link rel="alternate" title="%s projects feeds" '.
'href="%s" type="text/x-opml" />'."\n",
$site_name, href(project=>undef, action=>"opml"));
}
if (defined $favicon) {
- print qq(<link rel="shortcut icon" href="$favicon" type="image/png" />\n);
+ print {$output_handler} qq(<link rel="shortcut icon" href="$favicon" type="image/png" />\n);
}
- print "</head>\n" .
+ print {$output_handler} "</head>\n" .
"<body>\n";
if ($site_header && -f $site_header) {
insert_file($site_header);
}
- print "<div class=\"page_header\">\n" .
+ print {$output_handler} "<div class=\"page_header\">\n" .
$cgi->a({-href => esc_url($logo_url),
-title => $logo_label},
qq(<img src="$logo" width="72" height="27" alt="git" class="logo"/>));
- print $cgi->a({-href => esc_url($home_link)}, $home_link_str) . " / ";
+ print {$output_handler} $cgi->a({-href => esc_url($home_link)}, $home_link_str) . " / ";
if (defined $project) {
- print $cgi->a({-href => href(action=>"summary")}, esc_html($project));
+ print {$output_handler} $cgi->a({-href => href(action=>"summary")}, esc_html($project));
if (defined $action) {
- print " / $action";
+ print {$output_handler} " / $action";
}
- print "\n";
+ print {$output_handler} "\n";
}
- print "</div>\n";
+ print {$output_handler} "</div>\n";
my $have_search = gitweb_check_feature('search');
if (defined $project && $have_search) {
@@ -3313,7 +3320,7 @@ EOF
if ($use_pathinfo) {
$action .= "/".esc_url($project);
}
- print $cgi->startform(-method => "get", -action => $action) .
+ print {$output_handler} $cgi->startform(-method => "get", -action => $action) .
"<div class=\"search\">\n" .
(!$use_pathinfo &&
$cgi->input({-name=>"p", -value=>$project, -type=>"hidden"}) . "\n") .
@@ -3336,11 +3343,11 @@ EOF
sub git_footer_html {
my $feed_class = 'rss_logo';
- print "<div class=\"page_footer\">\n";
+ print {$output_handler} "<div class=\"page_footer\">\n";
if (defined $project) {
my $descr = git_get_project_description($project);
if (defined $descr) {
- print "<div class=\"page_footer_text\">" . esc_html($descr) . "</div>\n";
+ print {$output_handler} "<div class=\"page_footer_text\">" . esc_html($descr) . "</div>\n";
}
my %href_params = get_feed_info();
@@ -3351,22 +3358,22 @@ sub git_footer_html {
foreach my $format qw(RSS Atom) {
$href_params{'action'} = lc($format);
- print $cgi->a({-href => href(%href_params),
+ print {$output_handler} $cgi->a({-href => href(%href_params),
-title => "$href_params{'-title'} $format feed",
-class => $feed_class}, $format)."\n";
}
} else {
- print $cgi->a({-href => href(project=>undef, action=>"opml"),
+ print {$output_handler} $cgi->a({-href => href(project=>undef, action=>"opml"),
-class => $feed_class}, "OPML") . " ";
- print $cgi->a({-href => href(project=>undef, action=>"project_index"),
+ print {$output_handler} $cgi->a({-href => href(project=>undef, action=>"project_index"),
-class => $feed_class}, "TXT") . "\n";
}
- print "</div>\n"; # class="page_footer"
+ print {$output_handler} "</div>\n"; # class="page_footer"
if (defined $t0 && gitweb_check_feature('timed')) {
- print "<div id=\"generating_info\">\n";
- print 'This page took '.
+ print {$output_handler} "<div id=\"generating_info\">\n";
+ print {$output_handler} 'This page took '.
'<span id="generating_time" class="time_span">'.
Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).
' seconds </span>'.
@@ -3375,26 +3382,26 @@ sub git_footer_html {
$number_of_git_cmds.
'</span> git commands '.
" to generate.\n";
- print "</div>\n"; # class="page_footer"
+ print {$output_handler} "</div>\n"; # class="page_footer"
}
if (-f $site_footer) {
insert_file($site_footer);
}
- print qq!<script type="text/javascript" src="$javascript"></script>\n!;
+ print {$output_handler} qq!<script type="text/javascript" src="$javascript"></script>\n!;
if ($action eq 'blame_incremental') {
- print qq!<script type="text/javascript">\n!.
+ print {$output_handler} qq!<script type="text/javascript">\n!.
qq!startBlame("!. href(action=>"blame_data", -replay=>1) .qq!",\n!.
qq! "!. href() .qq!");\n!.
qq!</script>\n!;
} elsif (gitweb_check_feature('javascript-actions')) {
- print qq!<script type="text/javascript">\n!.
+ print {$output_handler} qq!<script type="text/javascript">\n!.
qq!window.onload = fixLinks;\n!.
qq!</script>\n!;
}
- print "</body>\n" .
+ print {$output_handler} "</body>\n" .
"</html>";
}
@@ -3416,6 +3423,14 @@ sub die_error {
my $error = shift || "Internal server error";
my $extra = shift;
+ # The output handlers for die_error need to be reset to STDOUT
+ # so that half the message isn't being output to random and
+ # half to STDOUT as expected. This is mainly for the benefit
+ # of using git_header_html() and git_footer_html() since those
+ # internaly use the indirect print handler.
+ $output_handler = *STDOUT;
+ $output_handler_bin = *STDOUT;
+
my %http_responses = (
400 => '400 Bad Request',
403 => '403 Forbidden',
@@ -3484,12 +3499,12 @@ sub git_print_page_nav {
$arg{$label}{'_href'} = $link;
}
- print "<div class=\"page_nav\">\n" .
+ print {$output_handler} "<div class=\"page_nav\">\n" .
(join " | ",
map { $_ eq $current ?
$_ : $cgi->a({-href => ($arg{$_}{_href} ? $arg{$_}{_href} : href(%{$arg{$_}}))}, "$_")
} @navs);
- print "<br/>\n$extra<br/>\n" .
+ print {$output_handler} "<br/>\n$extra<br/>\n" .
"</div>\n";
}
@@ -3530,14 +3545,14 @@ sub git_print_header_div {
$args{'hash'} = $hash if $hash;
$args{'hash_base'} = $hash_base if $hash_base;
- print "<div class=\"header\">\n" .
+ print {$output_handler} "<div class=\"header\">\n" .
$cgi->a({-href => href(%args), -class => "title"},
$title ? $title : $action) .
"\n</div>\n";
}
sub print_local_time {
- print get_local_time(@_);
+ print {$output_handler} get_local_time(@_);
}
sub get_local_time {
@@ -3562,11 +3577,11 @@ sub git_print_authorship {
my $author = $co->{'author_name'};
my %ad = parse_date($co->{'author_epoch'}, $co->{'author_tz'});
- print "<$tag class=\"author_date\">" .
+ print {$output_handler} "<$tag class=\"author_date\">" .
format_search_author($author, "author", esc_html($author)) .
" [$ad{'rfc2822'}";
- print_local_time(%ad) if ($opts{-localtime});
- print "]" . git_get_avatar($co->{'author_email'}, -pad_before => 1)
+ print {$output_handler} get_local_time(%ad) if ($opts{-localtime});
+ print {$output_handler} "]" . git_get_avatar($co->{'author_email'}, -pad_before => 1)
. "</$tag>\n";
}
@@ -3582,7 +3597,7 @@ sub git_print_authorship_rows {
@people = ('author', 'committer') unless @people;
foreach my $who (@people) {
my %wd = parse_date($co->{"${who}_epoch"}, $co->{"${who}_tz"});
- print "<tr><td>$who</td><td>" .
+ print {$output_handler} "<tr><td>$who</td><td>" .
format_search_author($co->{"${who}_name"}, $who,
esc_html($co->{"${who}_name"})) . " " .
format_search_author($co->{"${who}_email"}, $who,
@@ -3592,8 +3607,8 @@ sub git_print_authorship_rows {
"</td></tr>\n" .
"<tr>" .
"<td></td><td> $wd{'rfc2822'}";
- print_local_time(%wd);
- print "</td>" .
+ print {$output_handler} get_local_time(%wd);
+ print {$output_handler} "</td>" .
"</tr>\n";
}
}
@@ -3603,11 +3618,10 @@ sub git_print_page_path {
my $type = shift;
my $hb = shift;
-
- print "<div class=\"page_path\">";
- print $cgi->a({-href => href(action=>"tree", hash_base=>$hb),
+ print {$output_handler} "<div class=\"page_path\">";
+ print {$output_handler} $cgi->a({-href => href(action=>"tree", hash_base=>$hb),
-title => 'tree root'}, to_utf8("[$project]"));
- print " / ";
+ print {$output_handler} " / ";
if (defined $name) {
my @dirname = split '/', $name;
my $basename = pop @dirname;
@@ -3615,25 +3629,25 @@ sub git_print_page_path {
foreach my $dir (@dirname) {
$fullname .= ($fullname ? '/' : '') . $dir;
- print $cgi->a({-href => href(action=>"tree", file_name=>$fullname,
+ print {$output_handler} $cgi->a({-href => href(action=>"tree", file_name=>$fullname,
hash_base=>$hb),
-title => $fullname}, esc_path($dir));
- print " / ";
+ print {$output_handler} " / ";
}
if (defined $type && $type eq 'blob') {
- print $cgi->a({-href => href(action=>"blob_plain", file_name=>$file_name,
+ print {$output_handler} $cgi->a({-href => href(action=>"blob_plain", file_name=>$file_name,
hash_base=>$hb),
-title => $name}, esc_path($basename));
} elsif (defined $type && $type eq 'tree') {
- print $cgi->a({-href => href(action=>"tree", file_name=>$file_name,
+ print {$output_handler} $cgi->a({-href => href(action=>"tree", file_name=>$file_name,
hash_base=>$hb),
-title => $name}, esc_path($basename));
- print " / ";
+ print {$output_handler} " / ";
} else {
- print esc_path($basename);
+ print {$output_handler} esc_path($basename);
}
}
- print "<br/></div>\n";
+ print {$output_handler} "<br/></div>\n";
}
sub git_print_log {
@@ -3657,7 +3671,7 @@ sub git_print_log {
$signoff = 1;
$empty = 0;
if (! $opts{'-remove_signoff'}) {
- print "<span class=\"signoff\">" . esc_html($line) . "</span><br/>\n";
+ print {$output_handler} "<span class=\"signoff\">" . esc_html($line) . "</span><br/>\n";
next;
} else {
# remove signoff lines
@@ -3676,12 +3690,12 @@ sub git_print_log {
$empty = 0;
}
- print format_log_line_html($line) . "<br/>\n";
+ print {$output_handler} format_log_line_html($line) . "<br/>\n";
}
if ($opts{'-final_empty_line'}) {
# end with single empty line
- print "<br/>\n" unless $empty;
+ print {$output_handler} "<br/>\n" unless $empty;
}
}
@@ -3754,12 +3768,12 @@ sub git_print_tree_entry {
# the mode of the entry, list is the name of the entry, an href,
# and link is the action links of the entry.
- print "<td class=\"mode\">" . mode_str($t->{'mode'}) . "</td>\n";
+ print {$output_handler} "<td class=\"mode\">" . mode_str($t->{'mode'}) . "</td>\n";
if (exists $t->{'size'}) {
- print "<td class=\"size\">$t->{'size'}</td>\n";
+ print {$output_handler} "<td class=\"size\">$t->{'size'}</td>\n";
}
if ($t->{'type'} eq "blob") {
- print "<td class=\"list\">" .
+ print {$output_handler} "<td class=\"list\">" .
$cgi->a({-href => href(action=>"blob", hash=>$t->{'hash'},
file_name=>"$basedir$t->{'name'}", %base_key),
-class => "list"}, esc_path($t->{'name'}));
@@ -3768,71 +3782,71 @@ sub git_print_tree_entry {
if ($link_target) {
my $norm_target = normalize_link_target($link_target, $basedir);
if (defined $norm_target) {
- print " -> " .
+ print {$output_handler} " -> " .
$cgi->a({-href => href(action=>"object", hash_base=>$hash_base,
file_name=>$norm_target),
-title => $norm_target}, esc_path($link_target));
} else {
- print " -> " . esc_path($link_target);
+ print {$output_handler} " -> " . esc_path($link_target);
}
}
}
- print "</td>\n";
- print "<td class=\"link\">";
- print $cgi->a({-href => href(action=>"blob", hash=>$t->{'hash'},
+ print {$output_handler} "</td>\n";
+ print {$output_handler} "<td class=\"link\">";
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$t->{'hash'},
file_name=>"$basedir$t->{'name'}", %base_key)},
"blob");
if ($have_blame) {
- print " | " .
+ print {$output_handler} " | " .
$cgi->a({-href => href(action=>"blame", hash=>$t->{'hash'},
file_name=>"$basedir$t->{'name'}", %base_key)},
"blame");
}
if (defined $hash_base) {
- print " | " .
+ print {$output_handler} " | " .
$cgi->a({-href => href(action=>"history", hash_base=>$hash_base,
hash=>$t->{'hash'}, file_name=>"$basedir$t->{'name'}")},
"history");
}
- print " | " .
+ print {$output_handler} " | " .
$cgi->a({-href => href(action=>"blob_plain", hash_base=>$hash_base,
file_name=>"$basedir$t->{'name'}")},
"raw");
- print "</td>\n";
+ print {$output_handler} "</td>\n";
} elsif ($t->{'type'} eq "tree") {
- print "<td class=\"list\">";
- print $cgi->a({-href => href(action=>"tree", hash=>$t->{'hash'},
+ print {$output_handler} "<td class=\"list\">";
+ print {$output_handler} $cgi->a({-href => href(action=>"tree", hash=>$t->{'hash'},
file_name=>"$basedir$t->{'name'}",
%base_key)},
esc_path($t->{'name'}));
- print "</td>\n";
- print "<td class=\"link\">";
- print $cgi->a({-href => href(action=>"tree", hash=>$t->{'hash'},
+ print {$output_handler} "</td>\n";
+ print {$output_handler} "<td class=\"link\">";
+ print {$output_handler} $cgi->a({-href => href(action=>"tree", hash=>$t->{'hash'},
file_name=>"$basedir$t->{'name'}",
%base_key)},
"tree");
if (defined $hash_base) {
- print " | " .
+ print {$output_handler} " | " .
$cgi->a({-href => href(action=>"history", hash_base=>$hash_base,
file_name=>"$basedir$t->{'name'}")},
"history");
}
- print "</td>\n";
+ print {$output_handler} "</td>\n";
} else {
# unknown object: we can only present history for it
# (this includes 'commit' object, i.e. submodule support)
- print "<td class=\"list\">" .
+ print {$output_handler} "<td class=\"list\">" .
esc_path($t->{'name'}) .
"</td>\n";
- print "<td class=\"link\">";
+ print {$output_handler} "<td class=\"link\">";
if (defined $hash_base) {
- print $cgi->a({-href => href(action=>"history",
+ print {$output_handler} $cgi->a({-href => href(action=>"history",
hash_base=>$hash_base,
file_name=>"$basedir$t->{'name'}")},
"history");
}
- print "</td>\n";
+ print {$output_handler} "</td>\n";
}
}
@@ -3879,13 +3893,13 @@ sub git_difftree_body {
my ($difftree, $hash, @parents) = @_;
my ($parent) = $parents[0];
my $have_blame = gitweb_check_feature('blame');
- print "<div class=\"list_head\">\n";
+ print {$output_handler} "<div class=\"list_head\">\n";
if ($#{$difftree} > 10) {
- print(($#{$difftree} + 1) . " files changed:\n");
+ print {$output_handler} (($#{$difftree} + 1) . " files changed:\n");
}
- print "</div>\n";
+ print {$output_handler} "</div>\n";
- print "<table class=\"" .
+ print {$output_handler} "<table class=\"" .
(@parents > 1 ? "combined " : "") .
"diff_tree\">\n";
@@ -3893,11 +3907,11 @@ sub git_difftree_body {
my $has_header = @$difftree && @parents > 1 && $action eq 'commitdiff';
if ($has_header) {
# table header
- print "<thead><tr>\n" .
+ print {$output_handler} "<thead><tr>\n" .
"<th></th><th></th>\n"; # filename, patchN link
for (my $i = 0; $i < @parents; $i++) {
my $par = $parents[$i];
- print "<th>" .
+ print {$output_handler} "<th>" .
$cgi->a({-href => href(action=>"commitdiff",
hash=>$hash, hash_parent=>$par),
-title => 'commitdiff to parent number ' .
@@ -3905,7 +3919,7 @@ sub git_difftree_body {
$i+1) .
" </th>\n";
}
- print "</tr></thead>\n<tbody>\n";
+ print {$output_handler} "</tr></thead>\n<tbody>\n";
}
my $alternate = 1;
@@ -3914,9 +3928,9 @@ sub git_difftree_body {
my $diff = parsed_difftree_line($line);
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
@@ -3927,14 +3941,14 @@ sub git_difftree_body {
if (!is_deleted($diff)) {
# file exists in the result (child) commit
- print "<td>" .
+ print {$output_handler} "<td>" .
$cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
file_name=>$diff->{'to_file'},
hash_base=>$hash),
-class => "list"}, esc_path($diff->{'to_file'})) .
"</td>\n";
} else {
- print "<td>" .
+ print {$output_handler} "<td>" .
esc_path($diff->{'to_file'}) .
"</td>\n";
}
@@ -3942,7 +3956,7 @@ sub git_difftree_body {
if ($action eq 'commitdiff') {
# link to patch
$patchno++;
- print "<td class=\"link\">" .
+ print {$output_handler} "<td class=\"link\">" .
$cgi->a({-href => "#patch$patchno"}, "patch") .
" | " .
"</td>\n";
@@ -3960,9 +3974,9 @@ sub git_difftree_body {
$not_deleted ||= ($status ne 'D');
if ($status eq 'A') {
- print "<td class=\"link\" align=\"right\"> | </td>\n";
+ print {$output_handler} "<td class=\"link\" align=\"right\"> | </td>\n";
} elsif ($status eq 'D') {
- print "<td class=\"link\">" .
+ print {$output_handler} "<td class=\"link\">" .
$cgi->a({-href => href(action=>"blob",
hash_base=>$hash,
hash=>$from_hash,
@@ -3971,11 +3985,11 @@ sub git_difftree_body {
" | </td>\n";
} else {
if ($diff->{'to_id'} eq $from_hash) {
- print "<td class=\"link nochange\">";
+ print {$output_handler} "<td class=\"link nochange\">";
} else {
- print "<td class=\"link\">";
+ print {$output_handler} "<td class=\"link\">";
}
- print $cgi->a({-href => href(action=>"blobdiff",
+ print {$output_handler} $cgi->a({-href => href(action=>"blobdiff",
hash=>$diff->{'to_id'},
hash_parent=>$from_hash,
hash_base=>$hash,
@@ -3987,24 +4001,24 @@ sub git_difftree_body {
}
}
- print "<td class=\"link\">";
+ print {$output_handler} "<td class=\"link\">";
if ($not_deleted) {
- print $cgi->a({-href => href(action=>"blob",
+ print {$output_handler} $cgi->a({-href => href(action=>"blob",
hash=>$diff->{'to_id'},
file_name=>$diff->{'to_file'},
hash_base=>$hash)},
"blob");
- print " | " if ($has_history);
+ print {$output_handler} " | " if ($has_history);
}
if ($has_history) {
- print $cgi->a({-href => href(action=>"history",
+ print {$output_handler} $cgi->a({-href => href(action=>"history",
file_name=>$diff->{'to_file'},
hash_base=>$hash)},
"history");
}
- print "</td>\n";
+ print {$output_handler} "</td>\n";
- print "</tr>\n";
+ print {$output_handler} "</tr>\n";
next; # instead of 'else' clause, to avoid extra indent
}
# else ordinary diff
@@ -4030,51 +4044,51 @@ sub git_difftree_body {
my $mode_chng = "<span class=\"file_status new\">[new $to_file_type";
$mode_chng .= " with mode: $to_mode_str" if $to_mode_str;
$mode_chng .= "]</span>";
- print "<td>";
- print $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
+ print {$output_handler} "<td>";
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
hash_base=>$hash, file_name=>$diff->{'file'}),
-class => "list"}, esc_path($diff->{'file'}));
- print "</td>\n";
- print "<td>$mode_chng</td>\n";
- print "<td class=\"link\">";
+ print {$output_handler} "</td>\n";
+ print {$output_handler} "<td>$mode_chng</td>\n";
+ print {$output_handler} "<td class=\"link\">";
if ($action eq 'commitdiff') {
# link to patch
$patchno++;
- print $cgi->a({-href => "#patch$patchno"}, "patch");
- print " | ";
+ print {$output_handler} $cgi->a({-href => "#patch$patchno"}, "patch");
+ print {$output_handler} " | ";
}
- print $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
hash_base=>$hash, file_name=>$diff->{'file'})},
"blob");
- print "</td>\n";
+ print {$output_handler} "</td>\n";
} elsif ($diff->{'status'} eq "D") { # deleted
my $mode_chng = "<span class=\"file_status deleted\">[deleted $from_file_type]</span>";
- print "<td>";
- print $cgi->a({-href => href(action=>"blob", hash=>$diff->{'from_id'},
+ print {$output_handler} "<td>";
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$diff->{'from_id'},
hash_base=>$parent, file_name=>$diff->{'file'}),
-class => "list"}, esc_path($diff->{'file'}));
- print "</td>\n";
- print "<td>$mode_chng</td>\n";
- print "<td class=\"link\">";
+ print {$output_handler} "</td>\n";
+ print {$output_handler} "<td>$mode_chng</td>\n";
+ print {$output_handler} "<td class=\"link\">";
if ($action eq 'commitdiff') {
# link to patch
$patchno++;
- print $cgi->a({-href => "#patch$patchno"}, "patch");
- print " | ";
+ print {$output_handler} $cgi->a({-href => "#patch$patchno"}, "patch");
+ print {$output_handler} " | ";
}
- print $cgi->a({-href => href(action=>"blob", hash=>$diff->{'from_id'},
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$diff->{'from_id'},
hash_base=>$parent, file_name=>$diff->{'file'})},
"blob") . " | ";
if ($have_blame) {
- print $cgi->a({-href => href(action=>"blame", hash_base=>$parent,
+ print {$output_handler} $cgi->a({-href => href(action=>"blame", hash_base=>$parent,
file_name=>$diff->{'file'})},
"blame") . " | ";
}
- print $cgi->a({-href => href(action=>"history", hash_base=>$parent,
+ print {$output_handler} $cgi->a({-href => href(action=>"history", hash_base=>$parent,
file_name=>$diff->{'file'})},
"history");
- print "</td>\n";
+ print {$output_handler} "</td>\n";
} elsif ($diff->{'status'} eq "M" || $diff->{'status'} eq "T") { # modified, or type changed
my $mode_chnge = "";
@@ -4092,39 +4106,39 @@ sub git_difftree_body {
}
$mode_chnge .= "]</span>\n";
}
- print "<td>";
- print $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
+ print {$output_handler} "<td>";
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
hash_base=>$hash, file_name=>$diff->{'file'}),
-class => "list"}, esc_path($diff->{'file'}));
- print "</td>\n";
- print "<td>$mode_chnge</td>\n";
- print "<td class=\"link\">";
+ print {$output_handler} "</td>\n";
+ print {$output_handler} "<td>$mode_chnge</td>\n";
+ print {$output_handler} "<td class=\"link\">";
if ($action eq 'commitdiff') {
# link to patch
$patchno++;
- print $cgi->a({-href => "#patch$patchno"}, "patch") .
+ print {$output_handler} $cgi->a({-href => "#patch$patchno"}, "patch") .
" | ";
} elsif ($diff->{'to_id'} ne $diff->{'from_id'}) {
# "commit" view and modified file (not onlu mode changed)
- print $cgi->a({-href => href(action=>"blobdiff",
+ print {$output_handler} $cgi->a({-href => href(action=>"blobdiff",
hash=>$diff->{'to_id'}, hash_parent=>$diff->{'from_id'},
hash_base=>$hash, hash_parent_base=>$parent,
file_name=>$diff->{'file'})},
"diff") .
" | ";
}
- print $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
hash_base=>$hash, file_name=>$diff->{'file'})},
"blob") . " | ";
if ($have_blame) {
- print $cgi->a({-href => href(action=>"blame", hash_base=>$hash,
+ print {$output_handler} $cgi->a({-href => href(action=>"blame", hash_base=>$hash,
file_name=>$diff->{'file'})},
"blame") . " | ";
}
- print $cgi->a({-href => href(action=>"history", hash_base=>$hash,
+ print {$output_handler} $cgi->a({-href => href(action=>"history", hash_base=>$hash,
file_name=>$diff->{'file'})},
"history");
- print "</td>\n";
+ print {$output_handler} "</td>\n";
} elsif ($diff->{'status'} eq "R" || $diff->{'status'} eq "C") { # renamed or copied
my %status_name = ('R' => 'moved', 'C' => 'copied');
@@ -4134,7 +4148,7 @@ sub git_difftree_body {
# mode also for directories, so we cannot use $to_mode_str
$mode_chng = sprintf(", mode: %04o", $to_mode_oct & 0777);
}
- print "<td>" .
+ print {$output_handler} "<td>" .
$cgi->a({-href => href(action=>"blob", hash_base=>$hash,
hash=>$diff->{'to_id'}, file_name=>$diff->{'to_file'}),
-class => "list"}, esc_path($diff->{'to_file'})) . "</td>\n" .
@@ -4147,35 +4161,35 @@ sub git_difftree_body {
if ($action eq 'commitdiff') {
# link to patch
$patchno++;
- print $cgi->a({-href => "#patch$patchno"}, "patch") .
+ print {$output_handler} $cgi->a({-href => "#patch$patchno"}, "patch") .
" | ";
} elsif ($diff->{'to_id'} ne $diff->{'from_id'}) {
# "commit" view and modified file (not only pure rename or copy)
- print $cgi->a({-href => href(action=>"blobdiff",
+ print {$output_handler} $cgi->a({-href => href(action=>"blobdiff",
hash=>$diff->{'to_id'}, hash_parent=>$diff->{'from_id'},
hash_base=>$hash, hash_parent_base=>$parent,
file_name=>$diff->{'to_file'}, file_parent=>$diff->{'from_file'})},
"diff") .
" | ";
}
- print $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash=>$diff->{'to_id'},
hash_base=>$parent, file_name=>$diff->{'to_file'})},
"blob") . " | ";
if ($have_blame) {
- print $cgi->a({-href => href(action=>"blame", hash_base=>$hash,
+ print {$output_handler} $cgi->a({-href => href(action=>"blame", hash_base=>$hash,
file_name=>$diff->{'to_file'})},
"blame") . " | ";
}
- print $cgi->a({-href => href(action=>"history", hash_base=>$hash,
+ print {$output_handler} $cgi->a({-href => href(action=>"history", hash_base=>$hash,
file_name=>$diff->{'to_file'})},
"history");
- print "</td>\n";
+ print {$output_handler} "</td>\n";
} # we should not encounter Unmerged (U) or Unknown (X) status
- print "</tr>\n";
+ print {$output_handler} "</tr>\n";
}
- print "</tbody>" if $has_header;
- print "</table>\n";
+ print {$output_handler} "</tbody>" if $has_header;
+ print {$output_handler} "</table>\n";
}
sub git_patchset_body {
@@ -4190,7 +4204,7 @@ sub git_patchset_body {
my $to_name;
my (%from, %to);
- print "<div class=\"patchset\">\n";
+ print {$output_handler} "<div class=\"patchset\">\n";
# skip to first patch
while ($patch_line = <$fd>) {
@@ -4218,7 +4232,7 @@ sub git_patchset_body {
# and parse raw git-diff line if needed
if (is_patch_split($diffinfo, { 'to_file' => $to_name })) {
# this is continuation of a split patch
- print "<div class=\"patch cont\">\n";
+ print {$output_handler} "<div class=\"patch cont\">\n";
} else {
# advance raw git-diff output if needed
$patch_idx++ if defined $diffinfo;
@@ -4230,7 +4244,7 @@ sub git_patchset_body {
# find which patch (using pathname of result) we are at now;
if ($is_combined) {
while ($to_name ne $diffinfo->{'to_file'}) {
- print "<div class=\"patch\" id=\"patch". ($patch_idx+1) ."\">\n" .
+ print {$output_handler} "<div class=\"patch\" id=\"patch". ($patch_idx+1) ."\">\n" .
format_diff_cc_simplified($diffinfo, @hash_parents) .
"</div>\n"; # class="patch"
@@ -4247,7 +4261,7 @@ sub git_patchset_body {
# this is first patch for raw difftree line with $patch_idx index
# we index @$difftree array from 0, but number patches from 1
- print "<div class=\"patch\" id=\"patch". ($patch_idx+1) ."\">\n";
+ print {$output_handler} "<div class=\"patch\" id=\"patch". ($patch_idx+1) ."\">\n";
}
# git diff header
@@ -4255,25 +4269,25 @@ sub git_patchset_body {
#assert($patch_line !~ m!$/$!) if DEBUG; # is chomp-ed
$patch_number++;
# print "git diff" header
- print format_git_diff_header_line($patch_line, $diffinfo,
+ print {$output_handler} format_git_diff_header_line($patch_line, $diffinfo,
\%from, \%to);
# print extended diff header
- print "<div class=\"diff extended_header\">\n";
+ print {$output_handler} "<div class=\"diff extended_header\">\n";
EXTENDED_HEADER:
while ($patch_line = <$fd>) {
chomp $patch_line;
last EXTENDED_HEADER if ($patch_line =~ m/^--- |^diff /);
- print format_extended_diff_header_line($patch_line, $diffinfo,
+ print {$output_handler} format_extended_diff_header_line($patch_line, $diffinfo,
\%from, \%to);
}
- print "</div>\n"; # class="diff extended_header"
+ print {$output_handler} "</div>\n"; # class="diff extended_header"
# from-file/to-file diff header
if (! $patch_line) {
- print "</div>\n"; # class="patch"
+ print {$output_handler} "</div>\n"; # class="patch"
last PATCH;
}
next PATCH if ($patch_line =~ m/^diff /);
@@ -4284,7 +4298,7 @@ sub git_patchset_body {
chomp $patch_line;
#assert($patch_line =~ m/^\+\+\+/) if DEBUG;
- print format_diff_from_to_header($last_patch_line, $patch_line,
+ print {$output_handler} format_diff_from_to_header($last_patch_line, $patch_line,
$diffinfo, \%from, \%to,
@hash_parents);
@@ -4295,11 +4309,11 @@ sub git_patchset_body {
next PATCH if ($patch_line =~ m/^diff /);
- print format_diff_line($patch_line, \%from, \%to);
+ print {$output_handler} format_diff_line($patch_line, \%from, \%to);
}
} continue {
- print "</div>\n"; # class="patch"
+ print {$output_handler} "</div>\n"; # class="patch"
}
# for compact combined (--cc) format, with chunk and patch simpliciaction
@@ -4311,7 +4325,7 @@ sub git_patchset_body {
$diffinfo = parsed_difftree_line($difftree->[$patch_idx]);
# generate anchor for "patch" links in difftree / whatchanged part
- print "<div class=\"patch\" id=\"patch". ($patch_idx+1) ."\">\n" .
+ print {$output_handler} "<div class=\"patch\" id=\"patch". ($patch_idx+1) ."\">\n" .
format_diff_cc_simplified($diffinfo, @hash_parents) .
"</div>\n"; # class="patch"
@@ -4320,13 +4334,13 @@ sub git_patchset_body {
if ($patch_number == 0) {
if (@hash_parents > 1) {
- print "<div class=\"diff nodifferences\">Trivial merge</div>\n";
+ print {$output_handler} "<div class=\"diff nodifferences\">Trivial merge</div>\n";
} else {
- print "<div class=\"diff nodifferences\">No differences found</div>\n";
+ print {$output_handler} "<div class=\"diff nodifferences\">No differences found</div>\n";
}
}
- print "</div>\n"; # class="patchset"
+ print {$output_handler} "</div>\n"; # class="patchset"
}
# . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
@@ -4375,7 +4389,7 @@ sub fill_project_list_info {
# print 'sort by' <th> element, generating 'sort by $name' replay link
# if that order is not selected
sub print_sort_th {
- print get_sort_th(@_);
+ print {$output_handler} get_sort_th(@_);
}
sub get_sort_th {
@@ -4428,20 +4442,20 @@ sub git_project_list_body {
}
}
my $cloud = git_populate_project_tagcloud(\%ctags);
- print git_show_project_tagcloud($cloud, 64);
+ print {$output_handler} git_show_project_tagcloud($cloud, 64);
}
- print "<table class=\"project_list\">\n";
+ print {$output_handler} "<table class=\"project_list\">\n";
unless ($no_header) {
- print "<tr>\n";
+ print {$output_handler} "<tr>\n";
if ($check_forks) {
- print "<th></th>\n";
+ print {$output_handler} "<th></th>\n";
}
- print_sort_th('project', $order, 'Project');
- print_sort_th('descr', $order, 'Description');
- print_sort_th('owner', $order, 'Owner');
- print_sort_th('age', $order, 'Last Change');
- print "<th></th>\n" . # for links
+ print {$output_handler} get_sort_th('project', $order, 'Project');
+ print {$output_handler} get_sort_th('descr', $order, 'Description');
+ print {$output_handler} get_sort_th('owner', $order, 'Owner');
+ print {$output_handler} get_sort_th('age', $order, 'Last Change');
+ print {$output_handler} "<th></th>\n" . # for links
"</tr>\n";
}
my $alternate = 1;
@@ -4461,26 +4475,26 @@ sub git_project_list_body {
}
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
if ($check_forks) {
- print "<td>";
+ print {$output_handler} "<td>";
if ($pr->{'forks'}) {
- print "<!-- $pr->{'forks'} -->\n";
- print $cgi->a({-href => href(project=>$pr->{'path'}, action=>"forks")}, "+");
+ print {$output_handler} "<!-- $pr->{'forks'} -->\n";
+ print {$output_handler} $cgi->a({-href => href(project=>$pr->{'path'}, action=>"forks")}, "+");
}
- print "</td>\n";
+ print {$output_handler} "</td>\n";
}
- print "<td>" . $cgi->a({-href => href(project=>$pr->{'path'}, action=>"summary"),
- -class => "list"}, esc_html($pr->{'path'})) . "</td>\n" .
+ print {$output_handler} "<td>" . $cgi->a({-href => href(project=>$pr->{'path'}, action=>"summary"),
+ -class => "list"}, esc_html($pr->{'path'})) ."</td>\n".
"<td>" . $cgi->a({-href => href(project=>$pr->{'path'}, action=>"summary"),
-class => "list", -title => $pr->{'descr_long'}},
esc_html($pr->{'descr'})) . "</td>\n" .
"<td><i>" . chop_and_escape_str($pr->{'owner'}, 15) . "</i></td>\n";
- print "<td class=\"". age_class($pr->{'age'}) . "\">" .
+ print {$output_handler} "<td class=\"". age_class($pr->{'age'}) . "\">" .
(defined $pr->{'age_string'} ? $pr->{'age_string'} : "No commits") . "</td>\n" .
"<td class=\"link\">" .
$cgi->a({-href => href(project=>$pr->{'path'}, action=>"summary")}, "summary") . " | " .
@@ -4492,14 +4506,14 @@ sub git_project_list_body {
"</tr>\n";
}
if (defined $extra) {
- print "<tr>\n";
+ print {$output_handler} "<tr>\n";
if ($check_forks) {
- print "<td></td>\n";
+ print {$output_handler} "<td></td>\n";
}
- print "<td colspan=\"5\">$extra</td>\n" .
+ print {$output_handler} "<td colspan=\"5\">$extra</td>\n" .
"</tr>\n";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
sub git_log_body {
@@ -4519,7 +4533,7 @@ sub git_log_body {
"<span class=\"age\">$co{'age_string'}</span>" .
esc_html($co{'title'}) . $ref,
$commit);
- print "<div class=\"title_text\">\n" .
+ print {$output_handler} "<div class=\"title_text\">\n" .
"<div class=\"log_link\">\n" .
$cgi->a({-href => href(action=>"commit", hash=>$commit)}, "commit") .
" | " .
@@ -4529,16 +4543,16 @@ sub git_log_body {
"<br/>\n" .
"</div>\n";
git_print_authorship(\%co, -tag => 'span');
- print "<br/>\n</div>\n";
+ print {$output_handler} "<br/>\n</div>\n";
- print "<div class=\"log_body\">\n";
+ print {$output_handler} "<div class=\"log_body\">\n";
git_print_log($co{'comment'}, -final_empty_line=> 1);
- print "</div>\n";
+ print {$output_handler} "</div>\n";
}
if ($extra) {
- print "<div class=\"page_nav\">\n";
- print "$extra\n";
- print "</div>\n";
+ print {$output_handler} "<div class=\"page_nav\">\n";
+ print {$output_handler} "$extra\n";
+ print {$output_handler} "</div>\n";
}
}
@@ -4549,41 +4563,41 @@ sub git_shortlog_body {
$from = 0 unless defined $from;
$to = $#{$commitlist} if (!defined $to || $#{$commitlist} < $to);
- print "<table class=\"shortlog\">\n";
+ print {$output_handler} "<table class=\"shortlog\">\n";
my $alternate = 1;
for (my $i = $from; $i <= $to; $i++) {
my %co = %{$commitlist->[$i]};
my $commit = $co{'id'};
my $ref = format_ref_marker($refs, $commit);
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
# git_summary() used print "<td><i>$co{'age_string'}</i></td>\n" .
- print "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
+ print {$output_handler} "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
format_author_html('td', \%co, 10) . "<td>";
- print format_subject_html($co{'title'}, $co{'title_short'},
+ print {$output_handler} format_subject_html($co{'title'}, $co{'title_short'},
href(action=>"commit", hash=>$commit), $ref);
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"<td class=\"link\">" .
$cgi->a({-href => href(action=>"commit", hash=>$commit)}, "commit") . " | " .
$cgi->a({-href => href(action=>"commitdiff", hash=>$commit)}, "commitdiff") . " | " .
$cgi->a({-href => href(action=>"tree", hash=>$commit, hash_base=>$commit)}, "tree");
my $snapshot_links = format_snapshot_links($commit);
if (defined $snapshot_links) {
- print " | " . $snapshot_links;
+ print {$output_handler} " | " . $snapshot_links;
}
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"</tr>\n";
}
if (defined $extra) {
- print "<tr>\n" .
+ print {$output_handler} "<tr>\n" .
"<td colspan=\"4\">$extra</td>\n" .
"</tr>\n";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
sub git_history_body {
@@ -4594,7 +4608,7 @@ sub git_history_body {
$from = 0 unless defined $from;
$to = $#{$commitlist} unless (defined $to && $to <= $#{$commitlist});
- print "<table class=\"history\">\n";
+ print {$output_handler} "<table class=\"history\">\n";
my $alternate = 1;
for (my $i = $from; $i <= $to; $i++) {
my %co = %{$commitlist->[$i]};
@@ -4606,18 +4620,18 @@ sub git_history_body {
my $ref = format_ref_marker($refs, $commit);
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
- print "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
+ print {$output_handler} "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
# shortlog: format_author_html('td', \%co, 10)
format_author_html('td', \%co, 15, 3) . "<td>";
# originally git_history used chop_str($co{'title'}, 50)
- print format_subject_html($co{'title'}, $co{'title_short'},
+ print {$output_handler} format_subject_html($co{'title'}, $co{'title_short'},
href(action=>"commit", hash=>$commit), $ref);
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"<td class=\"link\">" .
$cgi->a({-href => href(action=>$ftype, hash_base=>$commit, file_name=>$file_name)}, $ftype) . " | " .
$cgi->a({-href => href(action=>"commitdiff", hash=>$commit)}, "commitdiff");
@@ -4627,7 +4641,7 @@ sub git_history_body {
my $blob_parent = git_get_hash_by_path($commit, $file_name);
if (defined $blob_current && defined $blob_parent &&
$blob_current ne $blob_parent) {
- print " | " .
+ print {$output_handler} " | " .
$cgi->a({-href => href(action=>"blobdiff",
hash=>$blob_current, hash_parent=>$blob_parent,
hash_base=>$hash_base, hash_parent_base=>$commit,
@@ -4635,15 +4649,15 @@ sub git_history_body {
"diff to current");
}
}
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"</tr>\n";
}
if (defined $extra) {
- print "<tr>\n" .
+ print {$output_handler} "<tr>\n" .
"<td colspan=\"4\">$extra</td>\n" .
"</tr>\n";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
sub git_tags_body {
@@ -4652,7 +4666,7 @@ sub git_tags_body {
$from = 0 unless defined $from;
$to = $#{$taglist} if (!defined $to || $#{$taglist} < $to);
- print "<table class=\"tags\">\n";
+ print {$output_handler} "<table class=\"tags\">\n";
my $alternate = 1;
for (my $i = $from; $i <= $to; $i++) {
my $entry = $taglist->[$i];
@@ -4663,50 +4677,50 @@ sub git_tags_body {
$comment_short = chop_str($comment, 30, 5);
}
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
if (defined $tag{'age'}) {
- print "<td><i>$tag{'age'}</i></td>\n";
+ print {$output_handler} "<td><i>$tag{'age'}</i></td>\n";
} else {
- print "<td></td>\n";
+ print {$output_handler} "<td></td>\n";
}
- print "<td>" .
+ print {$output_handler} "<td>" .
$cgi->a({-href => href(action=>$tag{'reftype'}, hash=>$tag{'refid'}),
-class => "list name"}, esc_html($tag{'name'})) .
"</td>\n" .
"<td>";
if (defined $comment) {
- print format_subject_html($comment, $comment_short,
+ print {$output_handler} format_subject_html($comment, $comment_short,
href(action=>"tag", hash=>$tag{'id'}));
}
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"<td class=\"selflink\">";
if ($tag{'type'} eq "tag") {
- print $cgi->a({-href => href(action=>"tag", hash=>$tag{'id'})}, "tag");
+ print {$output_handler} $cgi->a({-href => href(action=>"tag", hash=>$tag{'id'})}, "tag");
} else {
- print " ";
+ print {$output_handler} " ";
}
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"<td class=\"link\">" . " | " .
$cgi->a({-href => href(action=>$tag{'reftype'}, hash=>$tag{'refid'})}, $tag{'reftype'});
if ($tag{'reftype'} eq "commit") {
- print " | " . $cgi->a({-href => href(action=>"shortlog", hash=>$tag{'fullname'})}, "shortlog") .
+ print {$output_handler} " | " . $cgi->a({-href => href(action=>"shortlog", hash=>$tag{'fullname'})}, "shortlog") .
" | " . $cgi->a({-href => href(action=>"log", hash=>$tag{'fullname'})}, "log");
} elsif ($tag{'reftype'} eq "blob") {
- print " | " . $cgi->a({-href => href(action=>"blob_plain", hash=>$tag{'refid'})}, "raw");
+ print {$output_handler} " | " . $cgi->a({-href => href(action=>"blob_plain", hash=>$tag{'refid'})}, "raw");
}
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"</tr>";
}
if (defined $extra) {
- print "<tr>\n" .
+ print {$output_handler} "<tr>\n" .
"<td colspan=\"5\">$extra</td>\n" .
"</tr>\n";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
sub git_heads_body {
@@ -4715,19 +4729,19 @@ sub git_heads_body {
$from = 0 unless defined $from;
$to = $#{$headlist} if (!defined $to || $#{$headlist} < $to);
- print "<table class=\"heads\">\n";
+ print {$output_handler} "<table class=\"heads\">\n";
my $alternate = 1;
for (my $i = $from; $i <= $to; $i++) {
my $entry = $headlist->[$i];
my %ref = %$entry;
my $curr = $ref{'id'} eq $head;
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
- print "<td><i>$ref{'age'}</i></td>\n" .
+ print {$output_handler} "<td><i>$ref{'age'}</i></td>\n" .
($curr ? "<td class=\"current_head\">" : "<td>") .
$cgi->a({-href => href(action=>"shortlog", hash=>$ref{'fullname'}),
-class => "list name"},esc_html($ref{'name'})) .
@@ -4740,11 +4754,11 @@ sub git_heads_body {
"</tr>";
}
if (defined $extra) {
- print "<tr>\n" .
+ print {$output_handler} "<tr>\n" .
"<td colspan=\"3\">$extra</td>\n" .
"</tr>\n";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
sub git_search_grep_body {
@@ -4752,7 +4766,7 @@ sub git_search_grep_body {
$from = 0 unless defined $from;
$to = $#{$commitlist} if (!defined $to || $#{$commitlist} < $to);
- print "<table class=\"commit_search\">\n";
+ print {$output_handler} "<table class=\"commit_search\">\n";
my $alternate = 1;
for (my $i = $from; $i <= $to; $i++) {
my %co = %{$commitlist->[$i]};
@@ -4761,12 +4775,12 @@ sub git_search_grep_body {
}
my $commit = $co{'id'};
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
- print "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
+ print {$output_handler} "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
format_author_html('td', \%co, 15, 5) .
"<td>" .
$cgi->a({-href => href(action=>"commit", hash=>$co{'id'}),
@@ -4786,25 +4800,25 @@ sub git_search_grep_body {
$match = esc_html($match);
$trail = esc_html($trail);
- print "$lead<span class=\"match\">$match</span>$trail<br />";
+ print {$output_handler} "$lead<span class=\"match\">$match</span>$trail<br />";
}
}
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"<td class=\"link\">" .
$cgi->a({-href => href(action=>"commit", hash=>$co{'id'})}, "commit") .
" | " .
$cgi->a({-href => href(action=>"commitdiff", hash=>$co{'id'})}, "commitdiff") .
" | " .
$cgi->a({-href => href(action=>"tree", hash=>$co{'tree'}, hash_base=>$co{'id'})}, "tree");
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"</tr>\n";
}
if (defined $extra) {
- print "<tr>\n" .
+ print {$output_handler} "<tr>\n" .
"<td colspan=\"3\">$extra</td>\n" .
"</tr>\n";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
## ======================================================================
@@ -4824,11 +4838,11 @@ sub git_project_list {
git_header_html();
if (-f $home_text) {
- print "<div class=\"index_include\">\n";
+ print {$output_handler} "<div class=\"index_include\">\n";
insert_file($home_text);
- print "</div>\n";
+ print {$output_handler} "</div>\n";
}
- print $cgi->startform(-method => "get") .
+ print {$output_handler} $cgi->startform(-method => "get") .
"<p class=\"projsearch\">Search:\n" .
$cgi->textfield(-name => "s", -value => $searchtext) . "\n" .
"</p>" .
@@ -4858,7 +4872,7 @@ sub git_forks {
sub git_project_index {
my @projects = git_get_projects_list($project);
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => 'text/plain',
-charset => 'utf-8',
-content_disposition => 'inline; filename="index.aux"');
@@ -4875,7 +4889,7 @@ sub git_project_index {
$path =~ s/ /\+/g;
$owner =~ s/ /\+/g;
- print "$path $owner\n";
+ print {$output_handler} "$path $owner\n";
}
}
@@ -4902,12 +4916,12 @@ sub git_summary {
git_header_html();
git_print_page_nav('summary','', $head);
- print "<div class=\"title\"> </div>\n";
- print "<table class=\"projects_list\">\n" .
+ print {$output_handler} "<div class=\"title\"> </div>\n";
+ print {$output_handler} "<table class=\"projects_list\">\n" .
"<tr id=\"metadata_desc\"><td>description</td><td>" . esc_html($descr) . "</td></tr>\n" .
"<tr id=\"metadata_owner\"><td>owner</td><td>" . esc_html($owner) . "</td></tr>\n";
if (defined $cd{'rfc2822'}) {
- print "<tr id=\"metadata_lchange\"><td>last change</td><td>$cd{'rfc2822'}</td></tr>\n";
+ print {$output_handler} "<tr id=\"metadata_lchange\"><td>last change</td><td>$cd{'rfc2822'}</td></tr>\n";
}
# use per project git URL list in $projectroot/$project/cloneurl
@@ -4917,7 +4931,7 @@ sub git_summary {
@url_list = map { "$_/$project" } @git_base_url_list unless @url_list;
foreach my $git_url (@url_list) {
next unless $git_url;
- print "<tr class=\"metadata_url\"><td>$url_tag</td><td>$git_url</td></tr>\n";
+ print {$output_handler} "<tr class=\"metadata_url\"><td>$url_tag</td><td>$git_url</td></tr>\n";
$url_tag = "";
}
@@ -4926,23 +4940,23 @@ sub git_summary {
if ($show_ctags) {
my $ctags = git_get_project_ctags($project);
my $cloud = git_populate_project_tagcloud($ctags);
- print "<tr id=\"metadata_ctags\"><td>Content tags:<br />";
- print "</td>\n<td>" unless %$ctags;
- print "<form action=\"$show_ctags\" method=\"post\"><input type=\"hidden\" name=\"p\" value=\"$project\" />Add: <input type=\"text\" name=\"t\" size=\"8\" /></form>";
- print "</td>\n<td>" if %$ctags;
- print git_show_project_tagcloud($cloud, 48);
- print "</td></tr>";
+ print {$output_handler} "<tr id=\"metadata_ctags\"><td>Content tags:<br />";
+ print {$output_handler} "</td>\n<td>" unless %$ctags;
+ print {$output_handler} "<form action=\"$show_ctags\" method=\"post\"><input type=\"hidden\" name=\"p\" value=\"$project\" />Add: <input type=\"text\" name=\"t\" size=\"8\" /></form>";
+ print {$output_handler} "</td>\n<td>" if %$ctags;
+ print {$output_handler} git_show_project_tagcloud($cloud, 48);
+ print {$output_handler} "</td></tr>";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
# If XSS prevention is on, we don't include README.html.
# TODO: Allow a readme in some safe format.
if (!$prevent_xss && -s "$projectroot/$project/README.html") {
- print "<div class=\"title\">readme</div>\n" .
+ print {$output_handler} "<div class=\"title\">readme</div>\n" .
"<div class=\"readme\">\n";
insert_file("$projectroot/$project/README.html");
- print "\n</div>\n"; # class="readme"
+ print {$output_handler} "\n</div>\n"; # class="readme"
}
# we need to request one more than 16 (0..15) to check if
@@ -4991,7 +5005,7 @@ sub git_tag {
}
git_print_header_div('commit', esc_html($tag{'name'}), $hash);
- print "<div class=\"title_text\">\n" .
+ print {$output_handler} "<div class=\"title_text\">\n" .
"<table class=\"object_header\">\n" .
"<tr>\n" .
"<td>object</td>\n" .
@@ -5003,15 +5017,15 @@ sub git_tag {
if (defined($tag{'author'})) {
git_print_authorship_rows(\%tag, 'author');
}
- print "</table>\n\n" .
+ print {$output_handler} "</table>\n\n" .
"</div>\n";
- print "<div class=\"page_body\">";
+ print {$output_handler} "<div class=\"page_body\">";
my $comment = $tag{'comment'};
foreach my $line (@$comment) {
chomp $line;
- print esc_html($line, -nbsp=>1) . "<br/>\n";
+ print {$output_handler} esc_html($line, -nbsp=>1) . "<br/>\n";
}
- print "</div>\n";
+ print {$output_handler} "</div>\n";
git_footer_html();
}
@@ -5062,21 +5076,23 @@ sub git_blame_common {
# incremental blame data returns early
if ($format eq 'data') {
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type=>"text/plain", -charset => "utf-8",
-status=> "200 OK");
local $| = 1; # output autoflush
- print while <$fd>;
+ while (<$fd>) {
+ print {$output_handler} $_;
+ }
close $fd
- or print "ERROR $!\n";
+ or die_error(500, "ERROR $!\n");
- print 'END';
+ print {$output_handler} 'END';
if (defined $t0 && gitweb_check_feature('timed')) {
- print ' '.
+ print {$output_handler} ' '.
Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).
' '.$number_of_git_cmds;
}
- print "\n";
+ print {$output_handler} "\n";
return;
}
@@ -5109,20 +5125,20 @@ sub git_blame_common {
# page body
if ($format eq 'incremental') {
- print "<noscript>\n<div class=\"error\"><center><b>\n".
+ print {$output_handler} "<noscript>\n<div class=\"error\"><center><b>\n".
"This page requires JavaScript to run.\n Use ".
$cgi->a({-href => href(action=>'blame',javascript=>0,-replay=>1)},
'this page').
" instead.\n".
"</b></center></div>\n</noscript>\n";
- print qq!<div id="progress_bar" style="width: 100%; background-color: yellow"></div>\n!;
+ print {$output_handler} qq!<div id="progress_bar" style="width: 100%; background-color: yellow"></div>\n!;
}
- print qq!<div class="page_body">\n!;
- print qq!<div id="progress_info">... / ...</div>\n!
+ print {$output_handler} qq!<div class="page_body">\n!;
+ print {$output_handler} qq!<div id="progress_info">... / ...</div>\n!
if ($format eq 'incremental');
- print qq!<table id="blame_table" class="blame" width="100%">\n!.
+ print {$output_handler} qq!<table id="blame_table" class="blame" width="100%">\n!.
#qq!<col width="5.5em" /><col width="2.5em" /><col width="*" />\n!.
qq!<thead>\n!.
qq!<tr><th>Commit</th><th>Line</th><th>Data</th></tr>\n!.
@@ -5143,12 +5159,12 @@ sub git_blame_common {
chomp $line;
$linenr++;
- print qq!<tr id="l$linenr" class="$color_class">!.
+ print {$output_handler} qq!<tr id="l$linenr" class="$color_class">!.
qq!<td class="sha1"><a href=""> </a></td>!.
qq!<td class="linenr">!.
qq!<a class="linenr" href="">$linenr</a></td>!;
- print qq!<td class="pre">! . esc_html($line) . "</td>\n";
- print qq!</tr>\n!;
+ print {$output_handler} qq!<td class="pre">! . esc_html($line) . "</td>\n";
+ print {$output_handler} qq!</tr>\n!;
}
} else { # porcelain, i.e. ordinary blame
@@ -5189,25 +5205,25 @@ sub git_blame_common {
$tr_class .= ' boundary' if (exists $meta->{'boundary'});
$tr_class .= ' no-previous' if ($meta->{'nprevious'} == 0);
$tr_class .= ' multiple-previous' if ($meta->{'nprevious'} > 1);
- print "<tr id=\"l$lineno\" class=\"$tr_class\">\n";
+ print {$output_handler} "<tr id=\"l$lineno\" class=\"$tr_class\">\n";
if ($group_size) {
- print "<td class=\"sha1\"";
- print " title=\"". esc_html($author) . ", $date\"";
- print " rowspan=\"$group_size\"" if ($group_size > 1);
- print ">";
- print $cgi->a({-href => href(action=>"commit",
+ print {$output_handler} "<td class=\"sha1\"";
+ print {$output_handler} " title=\"". esc_html($author) . ", $date\"";
+ print {$output_handler} " rowspan=\"$group_size\"" if ($group_size > 1);
+ print {$output_handler} ">";
+ print {$output_handler} $cgi->a({-href => href(action=>"commit",
hash=>$full_rev,
file_name=>$file_name)},
esc_html($short_rev));
if ($group_size >= 2) {
my @author_initials = ($author =~ /\b([[:upper:]])\B/g);
if (@author_initials) {
- print "<br />" .
+ print {$output_handler} "<br />" .
esc_html(join('', @author_initials));
# or join('.', ...)
}
}
- print "</td>\n";
+ print {$output_handler} "</td>\n";
}
# 'previous' <sha1 of parent commit> <filename at commit>
if (exists $meta->{'previous'} &&
@@ -5224,23 +5240,23 @@ sub git_blame_common {
my $blamed = href(action => 'blame',
file_name => $linenr_filename,
hash_base => $linenr_commit);
- print "<td class=\"linenr\">";
- print $cgi->a({ -href => "$blamed#l$orig_lineno",
+ print {$output_handler} "<td class=\"linenr\">";
+ print {$output_handler} $cgi->a({ -href => "$blamed#l$orig_lineno",
-class => "linenr" },
esc_html($lineno));
- print "</td>";
- print "<td class=\"pre\">" . esc_html($data) . "</td>\n";
- print "</tr>\n";
+ print {$output_handler} "</td>";
+ print {$output_handler} "<td class=\"pre\">" . esc_html($data) . "</td>\n";
+ print {$output_handler} "</tr>\n";
} # end while
}
# footer
- print "</tbody>\n".
+ print {$output_handler} "</tbody>\n".
"</table>\n"; # class="blame"
- print "</div>\n"; # class="blame_body"
+ print {$output_handler} "</div>\n"; # class="blame_body"
close $fd
- or print "Reading blob failed\n";
+ or print {$output_handler} "Reading blob failed\n";
git_footer_html();
}
@@ -5323,16 +5339,17 @@ sub git_blob_plain {
my $sandbox = $prevent_xss &&
$type !~ m!^(?:text/plain|image/(?:gif|png|jpeg))$!;
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => $type,
-expires => $expires,
-content_disposition =>
($sandbox ? 'attachment' : 'inline')
. '; filename="' . $save_as . '"');
local $/ = undef;
- binmode STDOUT, ':raw';
- print <$fd>;
- binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi
+
+ binmode $output_handler_bin, ':raw';
+ print {$output_handler_bin} <$fd>;
+ binmode $output_handler_bin, ':utf8'; # as set at the beginning of gitweb.cgi
close $fd;
}
@@ -5391,18 +5408,18 @@ sub git_blob {
git_print_page_nav('','', $hash_base,$co{'tree'},$hash_base, $formats_nav);
git_print_header_div('commit', esc_html($co{'title'}), $hash_base);
} else {
- print "<div class=\"page_nav\">\n" .
+ print {$output_handler} "<div class=\"page_nav\">\n" .
"<br/><br/></div>\n" .
"<div class=\"title\">$hash</div>\n";
}
git_print_page_path($file_name, "blob", $hash_base);
- print "<div class=\"page_body\">\n";
+ print {$output_handler} "<div class=\"page_body\">\n";
if ($mimetype =~ m!^image/!) {
- print qq!<img type="$mimetype"!;
+ print {$output_handler} qq!<img type="$mimetype"!;
if ($file_name) {
- print qq! alt="$file_name" title="$file_name"!;
+ print {$output_handler} qq! alt="$file_name" title="$file_name"!;
}
- print qq! src="! .
+ print {$output_handler} qq! src="! .
href(action=>"blob_plain", hash=>$hash,
hash_base=>$hash_base, file_name=>$file_name) .
qq!" />\n!;
@@ -5412,14 +5429,14 @@ sub git_blob {
chomp $line;
$nr++;
$line = untabify($line);
- printf "<div class=\"pre\"><a id=\"l%i\" href=\"" . href(-replay => 1)
+ printf {$output_handler} "<div class=\"pre\"><a id=\"l%i\" href=\"" . href(-replay => 1)
. "#l%i\" class=\"linenr\">%4i</a> %s</div>\n",
$nr, $nr, $nr, esc_html($line, -nbsp=>1);
}
}
close $fd
- or print "Reading blob failed.\n";
- print "</div>";
+ or print {$output_handler} "Reading blob failed.\n";
+ print {$output_handler} "</div>";
git_footer_html();
}
@@ -5474,9 +5491,9 @@ sub git_tree {
git_print_header_div('commit', esc_html($co{'title'}) . $ref, $hash_base);
} else {
undef $hash_base;
- print "<div class=\"page_nav\">\n";
- print "<br/><br/></div>\n";
- print "<div class=\"title\">$hash</div>\n";
+ print {$output_handler} "<div class=\"page_nav\">\n";
+ print {$output_handler} "<br/><br/></div>\n";
+ print {$output_handler} "<div class=\"title\">$hash</div>\n";
}
if (defined $file_name) {
$basedir = $file_name;
@@ -5485,16 +5502,16 @@ sub git_tree {
}
git_print_page_path($file_name, 'tree', $hash_base);
}
- print "<div class=\"page_body\">\n";
- print "<table class=\"tree\">\n";
+ print {$output_handler} "<div class=\"page_body\">\n";
+ print {$output_handler} "<table class=\"tree\">\n";
my $alternate = 1;
# '..' (top directory) link if possible
if (defined $hash_base &&
defined $file_name && $file_name =~ m![^/]+$!) {
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
@@ -5502,33 +5519,33 @@ sub git_tree {
$up =~ s!/?[^/]+$!!;
undef $up unless $up;
# based on git_print_tree_entry
- print '<td class="mode">' . mode_str('040000') . "</td>\n";
- print '<td class="size"> </td>'."\n" if $show_sizes;
- print '<td class="list">';
- print $cgi->a({-href => href(action=>"tree",
+ print {$output_handler} '<td class="mode">' . mode_str('040000') . "</td>\n";
+ print {$output_handler} '<td class="size"> </td>'."\n" if $show_sizes;
+ print {$output_handler} '<td class="list">';
+ print {$output_handler} $cgi->a({-href => href(action=>"tree",
hash_base=>$hash_base,
file_name=>$up)},
"..");
- print "</td>\n";
- print "<td class=\"link\"></td>\n";
+ print {$output_handler} "</td>\n";
+ print {$output_handler} "<td class=\"link\"></td>\n";
- print "</tr>\n";
+ print {$output_handler} "</tr>\n";
}
foreach my $line (@entries) {
my %t = parse_ls_tree_line($line, -z => 1, -l => $show_sizes);
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
git_print_tree_entry(\%t, $basedir, $hash_base, $have_blame);
- print "</tr>\n";
+ print {$output_handler} "</tr>\n";
}
- print "</table>\n" .
+ print {$output_handler} "</table>\n" .
"</div>";
git_footer_html();
}
@@ -5605,16 +5622,16 @@ sub git_snapshot {
}
$filename =~ s/(["\\])/\\$1/g;
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => $known_snapshot_formats{$format}{'type'},
-content_disposition => 'inline; filename="' . $filename . '"',
-status => '200 OK');
open my $fd, "-|", $cmd
or die_error(500, "Execute git-archive failed");
- binmode STDOUT, ':raw';
- print <$fd>;
- binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi
+ binmode $output_handler_bin, ':raw';
+ print {$output_handler_bin} <$fd>;
+ binmode $output_handler_bin, ':utf8'; # as set at the beginning of gitweb.cgi
close $fd;
}
@@ -5765,11 +5782,11 @@ sub git_commit {
} else {
git_print_header_div('tree', esc_html($co{'title'}) . $ref, $co{'tree'}, $hash);
}
- print "<div class=\"title_text\">\n" .
+ print {$output_handler} "<div class=\"title_text\">\n" .
"<table class=\"object_header\">\n";
git_print_authorship_rows(\%co);
- print "<tr><td>commit</td><td class=\"sha1\">$co{'id'}</td></tr>\n";
- print "<tr>" .
+ print {$output_handler} "<tr><td>commit</td><td class=\"sha1\">$co{'id'}</td></tr>\n";
+ print {$output_handler} "<tr>" .
"<td>tree</td>" .
"<td class=\"sha1\">" .
$cgi->a({-href => href(action=>"tree", hash=>$co{'tree'}, hash_base=>$hash),
@@ -5780,13 +5797,13 @@ sub git_commit {
"tree");
my $snapshot_links = format_snapshot_links($hash);
if (defined $snapshot_links) {
- print " | " . $snapshot_links;
+ print {$output_handler} " | " . $snapshot_links;
}
- print "</td>" .
+ print {$output_handler} "</td>" .
"</tr>\n";
foreach my $par (@$parents) {
- print "<tr>" .
+ print {$output_handler} "<tr>" .
"<td>parent</td>" .
"<td class=\"sha1\">" .
$cgi->a({-href => href(action=>"commit", hash=>$par),
@@ -5799,12 +5816,12 @@ sub git_commit {
"</td>" .
"</tr>\n";
}
- print "</table>".
+ print {$output_handler} "</table>".
"</div>\n";
- print "<div class=\"page_body\">\n";
+ print {$output_handler} "<div class=\"page_body\">\n";
git_print_log($co{'comment'});
- print "</div>\n";
+ print {$output_handler} "</div>\n";
git_difftree_body(\@difftree, $hash, @$parents);
@@ -5852,7 +5869,7 @@ sub git_object {
die_error(400, "Not enough information to find object");
}
- print $cgi->redirect(-uri => href(action=>$type, -full=>1,
+ print {$output_handler} $cgi->redirect(-uri => href(action=>$type, -full=>1,
hash=>$hash, hash_base=>$hash_base,
file_name=>$file_name),
-status => '302 Found');
@@ -5943,23 +5960,23 @@ sub git_blobdiff {
git_print_page_nav('','', $hash_base,$co{'tree'},$hash_base, $formats_nav);
git_print_header_div('commit', esc_html($co{'title'}), $hash_base);
} else {
- print "<div class=\"page_nav\"><br/>$formats_nav<br/></div>\n";
- print "<div class=\"title\">$hash vs $hash_parent</div>\n";
+ print {$output_handler} "<div class=\"page_nav\"><br/>$formats_nav<br/></div>\n";
+ print {$output_handler} "<div class=\"title\">$hash vs $hash_parent</div>\n";
}
if (defined $file_name) {
git_print_page_path($file_name, "blob", $hash_base);
} else {
- print "<div class=\"page_path\"></div>\n";
+ print {$output_handler} "<div class=\"page_path\"></div>\n";
}
} elsif ($format eq 'plain') {
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => 'text/plain',
-charset => 'utf-8',
-expires => $expires,
-content_disposition => 'inline; filename="' . "$file_name" . '.patch"');
- print "X-Git-Url: " . $cgi->self_url() . "\n\n";
+ print {$output_handler} "X-Git-Url: " . $cgi->self_url() . "\n\n";
} else {
die_error(400, "Unknown blobdiff format");
@@ -5967,12 +5984,12 @@ sub git_blobdiff {
# patch
if ($format eq 'html') {
- print "<div class=\"page_body\">\n";
+ print {$output_handler} "<div class=\"page_body\">\n";
git_patchset_body($fd, [ \%diffinfo ], $hash_base, $hash_parent_base);
close $fd;
- print "</div>\n"; # class="page_body"
+ print {$output_handler} "</div>\n"; # class="page_body"
git_footer_html();
} else {
@@ -5980,12 +5997,12 @@ sub git_blobdiff {
$line =~ s!a/($hash|$hash_parent)!'a/'.esc_path($diffinfo{'from_file'})!eg;
$line =~ s!b/($hash|$hash_parent)!'b/'.esc_path($diffinfo{'to_file'})!eg;
- print $line;
+ print {$output_handler} $line;
last if $line =~ m!^\+\+\+!;
}
local $/ = undef;
- print <$fd>;
+ print {$output_handler} <$fd>;
close $fd;
}
}
@@ -6148,16 +6165,16 @@ sub git_commitdiff {
git_header_html(undef, $expires);
git_print_page_nav('commitdiff','', $hash,$co{'tree'},$hash, $formats_nav);
git_print_header_div('commit', esc_html($co{'title'}) . $ref, $hash);
- print "<div class=\"title_text\">\n" .
+ print {$output_handler} "<div class=\"title_text\">\n" .
"<table class=\"object_header\">\n";
git_print_authorship_rows(\%co);
- print "</table>".
+ print {$output_handler} "</table>".
"</div>\n";
- print "<div class=\"page_body\">\n";
+ print {$output_handler} "<div class=\"page_body\">\n";
if (@{$co{'comment'}} > 1) {
- print "<div class=\"log\">\n";
+ print {$output_handler} "<div class=\"log\">\n";
git_print_log($co{'comment'}, -final_empty_line=> 1, -remove_title => 1);
- print "</div>\n"; # class="log"
+ print {$output_handler} "</div>\n"; # class="log"
}
} elsif ($format eq 'plain') {
@@ -6165,27 +6182,27 @@ sub git_commitdiff {
my $tagname = git_get_rev_name_tags($hash);
my $filename = basename($project) . "-$hash.patch";
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => 'text/plain',
-charset => 'utf-8',
-expires => $expires,
-content_disposition => 'inline; filename="' . "$filename" . '"');
my %ad = parse_date($co{'author_epoch'}, $co{'author_tz'});
- print "From: " . to_utf8($co{'author'}) . "\n";
- print "Date: $ad{'rfc2822'} ($ad{'tz_local'})\n";
- print "Subject: " . to_utf8($co{'title'}) . "\n";
+ print {$output_handler} "From: " . to_utf8($co{'author'}) . "\n";
+ print {$output_handler} "Date: $ad{'rfc2822'} ($ad{'tz_local'})\n";
+ print {$output_handler} "Subject: " . to_utf8($co{'title'}) . "\n";
- print "X-Git-Tag: $tagname\n" if $tagname;
- print "X-Git-Url: " . $cgi->self_url() . "\n\n";
+ print {$output_handler} "X-Git-Tag: $tagname\n" if $tagname;
+ print {$output_handler} "X-Git-Url: " . $cgi->self_url() . "\n\n";
foreach my $line (@{$co{'comment'}}) {
- print to_utf8($line) . "\n";
+ print {$output_handler} to_utf8($line) . "\n";
}
- print "---\n\n";
+ print {$output_handler} "---\n\n";
} elsif ($format eq 'patch') {
my $filename = basename($project) . "-$hash.patch";
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => 'text/plain',
-charset => 'utf-8',
-expires => $expires,
@@ -6198,24 +6215,24 @@ sub git_commitdiff {
$hash_parent eq '-c' || $hash_parent eq '--cc';
git_difftree_body(\@difftree, $hash,
$use_parents ? @{$co{'parents'}} : $hash_parent);
- print "<br/>\n";
+ print {$output_handler} "<br/>\n";
git_patchset_body($fd, \@difftree, $hash,
$use_parents ? @{$co{'parents'}} : $hash_parent);
close $fd;
- print "</div>\n"; # class="page_body"
+ print {$output_handler} "</div>\n"; # class="page_body"
git_footer_html();
} elsif ($format eq 'plain') {
local $/ = undef;
- print <$fd>;
+ print {$output_handler} <$fd>;
close $fd
- or print "Reading git-diff-tree failed\n";
+ or print {$output_handler} "Reading git-diff-tree failed\n";
} elsif ($format eq 'patch') {
local $/ = undef;
- print <$fd>;
+ print {$output_handler} <$fd>;
close $fd
- or print "Reading git-format-patch failed\n";
+ or print {$output_handler} "Reading git-format-patch failed\n";
}
}
@@ -6318,7 +6335,7 @@ sub git_search {
git_print_page_nav('','', $hash,$co{'tree'},$hash);
git_print_header_div('commit', esc_html($co{'title'}), $hash);
- print "<table class=\"pickaxe search\">\n";
+ print {$output_handler} "<table class=\"pickaxe search\">\n";
my $alternate = 1;
local $/ = "\n";
open my $fd, '-|', git_cmd(), '--no-pager', 'log', @diff_opts,
@@ -6334,24 +6351,24 @@ sub git_search {
if (defined $set{'commit'}) {
# finish previous commit
if (%co) {
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"<td class=\"link\">" .
$cgi->a({-href => href(action=>"commit", hash=>$co{'id'})}, "commit") .
" | " .
$cgi->a({-href => href(action=>"tree", hash=>$co{'tree'}, hash_base=>$co{'id'})}, "tree");
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"</tr>\n";
}
if ($alternate) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
$alternate ^= 1;
%co = parse_commit($set{'commit'});
my $author = chop_and_escape_str($co{'author_name'}, 15, 5);
- print "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
+ print {$output_handler} "<td title=\"$co{'age_string_age'}\"><i>$co{'age_string_date'}</i></td>\n" .
"<td><i>$author</i></td>\n" .
"<td>" .
$cgi->a({-href => href(action=>"commit", hash=>$co{'id'}),
@@ -6360,7 +6377,7 @@ sub git_search {
} elsif (defined $set{'to_id'}) {
next if ($set{'to_id'} =~ m/^0{40}$/);
- print $cgi->a({-href => href(action=>"blob", hash_base=>$co{'id'},
+ print {$output_handler} $cgi->a({-href => href(action=>"blob", hash_base=>$co{'id'},
hash=>$set{'to_id'}, file_name=>$set{'to_file'}),
-class => "list"},
"<span class=\"match\">" . esc_path($set{'file'}) . "</span>") .
@@ -6371,23 +6388,23 @@ sub git_search {
# finish last commit (warning: repetition!)
if (%co) {
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"<td class=\"link\">" .
$cgi->a({-href => href(action=>"commit", hash=>$co{'id'})}, "commit") .
" | " .
$cgi->a({-href => href(action=>"tree", hash=>$co{'tree'}, hash_base=>$co{'id'})}, "tree");
- print "</td>\n" .
+ print {$output_handler} "</td>\n" .
"</tr>\n";
}
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
if ($searchtype eq 'grep') {
git_print_page_nav('','', $hash,$co{'tree'},$hash);
git_print_header_div('commit', esc_html($co{'title'}), $hash);
- print "<table class=\"grep_search\">\n";
+ print {$output_handler} "<table class=\"grep_search\">\n";
my $alternate = 1;
my $matches = 0;
local $/ = "\n";
@@ -6406,21 +6423,21 @@ sub git_search {
(undef, $file, $lno, $ltext) = split(/:/, $line, 4);
}
if ($file ne $lastfile) {
- $lastfile and print "</td></tr>\n";
+ $lastfile and print {$output_handler} "</td></tr>\n";
if ($alternate++) {
- print "<tr class=\"dark\">\n";
+ print {$output_handler} "<tr class=\"dark\">\n";
} else {
- print "<tr class=\"light\">\n";
+ print {$output_handler} "<tr class=\"light\">\n";
}
- print "<td class=\"list\">".
+ print {$output_handler} "<td class=\"list\">".
$cgi->a({-href => href(action=>"blob", hash=>$co{'hash'},
file_name=>"$file"),
-class => "list"}, esc_path($file));
- print "</td><td>\n";
+ print {$output_handler} "</td><td>\n";
$lastfile = $file;
}
if ($binary) {
- print "<div class=\"binary\">Binary file</div>\n";
+ print {$output_handler} "<div class=\"binary\">Binary file</div>\n";
} else {
$ltext = untabify($ltext);
if ($ltext =~ m/^(.*)($search_regexp)(.*)$/i) {
@@ -6432,7 +6449,7 @@ sub git_search {
} else {
$ltext = esc_html($ltext, -nbsp=>1);
}
- print "<div class=\"pre\">" .
+ print {$output_handler} "<div class=\"pre\">" .
$cgi->a({-href => href(action=>"blob", hash=>$co{'hash'},
file_name=>"$file").'#l'.$lno,
-class => "linenr"}, sprintf('%4i', $lno))
@@ -6440,16 +6457,16 @@ sub git_search {
}
}
if ($lastfile) {
- print "</td></tr>\n";
+ print {$output_handler} "</td></tr>\n";
if ($matches > 1000) {
- print "<div class=\"diff nodifferences\">Too many matches, listing trimmed</div>\n";
+ print {$output_handler} "<div class=\"diff nodifferences\">Too many matches, listing trimmed</div>\n";
}
} else {
- print "<div class=\"diff nodifferences\">No matches found</div>\n";
+ print {$output_handler} "<div class=\"diff nodifferences\">No matches found</div>\n";
}
close $fd;
- print "</table>\n";
+ print {$output_handler} "</table>\n";
}
git_footer_html();
}
@@ -6457,7 +6474,7 @@ sub git_search {
sub git_search_help {
git_header_html();
git_print_page_nav('','', $hash,$hash,$hash);
- print <<EOT;
+ print {$output_handler} <<EOT;
<p><strong>Pattern</strong> is by default a normal string that is matched precisely (but without
regard to case, except in the case of pickaxe). However, when you check the <em>re</em> checkbox,
the pattern entered is recognized as the POSIX extended
@@ -6469,7 +6486,7 @@ insensitive).</p>
EOT
my $have_grep = gitweb_check_feature('grep');
if ($have_grep) {
- print <<EOT;
+ print {$output_handler} <<EOT;
<dt><b>grep</b></dt>
<dd>All files in the currently selected tree (HEAD unless you are explicitly browsing
a different one) are searched for the given pattern. On large trees, this search can take
@@ -6478,7 +6495,7 @@ due to git-grep peculiarity, currently if regexp mode is turned off, the matches
case-sensitive.</dd>
EOT
}
- print <<EOT;
+ print {$output_handler} <<EOT;
<dt><b>author</b></dt>
<dd>Name and e-mail of the change author and date of birth of the patch will be scanned for the given pattern.</dd>
<dt><b>committer</b></dt>
@@ -6486,7 +6503,7 @@ EOT
EOT
my $have_pickaxe = gitweb_check_feature('pickaxe');
if ($have_pickaxe) {
- print <<EOT;
+ print {$output_handler} <<EOT;
<dt><b>pickaxe</b></dt>
<dd>All commits that caused the string to appear or disappear from any file (changes that
added, removed or "modified" the string) will be listed. This search can take a while and
@@ -6494,7 +6511,7 @@ takes a lot of strain on the server, so please use it wisely. Note that since yo
interested even in changes just changing the case as well, this search is case sensitive.</dd>
EOT
}
- print "</dl>\n";
+ print {$output_handler} "</dl>\n";
git_footer_html();
}
@@ -6541,7 +6558,7 @@ sub git_feed {
$since = Time::ParseDate::parsedate($if_modified, GMT => 1);
}
if (defined $since && $latest_epoch <= $since) {
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => $content_type,
-charset => 'utf-8',
-last_modified => $latest_date{'rfc2822'},
@@ -6549,12 +6566,12 @@ sub git_feed {
return;
}
}
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => $content_type,
-charset => 'utf-8',
-last_modified => $latest_date{'rfc2822'});
} else {
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => $content_type,
-charset => 'utf-8');
}
@@ -6598,13 +6615,13 @@ sub git_feed {
} else {
$alt_url = href(-full=>1, action=>"summary");
}
- print qq!<?xml version="1.0" encoding="utf-8"?>\n!;
+ print {$output_handler} qq!<?xml version="1.0" encoding="utf-8"?>\n!;
if ($format eq 'rss') {
- print <<XML;
+ print {$output_handler} <<XML;
<rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
<channel>
XML
- print "<title>$title</title>\n" .
+ print {$output_handler} "<title>$title</title>\n" .
"<link>$alt_url</link>\n" .
"<description>$descr</description>\n" .
"<language>en</language>\n" .
@@ -6614,22 +6631,22 @@ XML
# prefer the logo to the favicon, since RSS
# doesn't allow both
my $img = esc_url($logo || $favicon);
- print "<image>\n" .
+ print {$output_handler} "<image>\n" .
"<url>$img</url>\n" .
"<title>$title</title>\n" .
"<link>$alt_url</link>\n" .
"</image>\n";
}
if (%latest_date) {
- print "<pubDate>$latest_date{'rfc2822'}</pubDate>\n";
- print "<lastBuildDate>$latest_date{'rfc2822'}</lastBuildDate>\n";
+ print {$output_handler} "<pubDate>$latest_date{'rfc2822'}</pubDate>\n";
+ print {$output_handler} "<lastBuildDate>$latest_date{'rfc2822'}</lastBuildDate>\n";
}
- print "<generator>gitweb v.$version/$git_version</generator>\n";
+ print {$output_handler} "<generator>gitweb v.$version/$git_version</generator>\n";
} elsif ($format eq 'atom') {
- print <<XML;
+ print {$output_handler} <<XML;
<feed xmlns="http://www.w3.org/2005/Atom">
XML
- print "<title>$title</title>\n" .
+ print {$output_handler} "<title>$title</title>\n" .
"<subtitle>$descr</subtitle>\n" .
'<link rel="alternate" type="text/html" href="' .
$alt_url . '" />' . "\n" .
@@ -6639,19 +6656,19 @@ XML
# use project owner for feed author
"<author><name>$owner</name></author>\n";
if (defined $favicon) {
- print "<icon>" . esc_url($favicon) . "</icon>\n";
+ print {$output_handler} "<icon>" . esc_url($favicon) . "</icon>\n";
}
if (defined $logo_url) {
# not twice as wide as tall: 72 x 27 pixels
- print "<logo>" . esc_url($logo) . "</logo>\n";
+ print {$output_handler} "<logo>" . esc_url($logo) . "</logo>\n";
}
if (! %latest_date) {
# dummy date to keep the feed valid until commits trickle in:
- print "<updated>1970-01-01T00:00:00Z</updated>\n";
+ print {$output_handler} "<updated>1970-01-01T00:00:00Z</updated>\n";
} else {
- print "<updated>$latest_date{'iso-8601'}</updated>\n";
+ print {$output_handler} "<updated>$latest_date{'iso-8601'}</updated>\n";
}
- print "<generator version='$version/$git_version'>gitweb</generator>\n";
+ print {$output_handler} "<generator version='$version/$git_version'>gitweb</generator>\n";
}
# contents
@@ -6676,7 +6693,7 @@ XML
# print element (entry, item)
my $co_url = href(-full=>1, action=>"commitdiff", hash=>$commit);
if ($format eq 'rss') {
- print "<item>\n" .
+ print {$output_handler} "<item>\n" .
"<title>" . esc_html($co{'title'}) . "</title>\n" .
"<author>" . esc_html($co{'author'}) . "</author>\n" .
"<pubDate>$cd{'rfc2822'}</pubDate>\n" .
@@ -6686,22 +6703,22 @@ XML
"<content:encoded>" .
"<![CDATA[\n";
} elsif ($format eq 'atom') {
- print "<entry>\n" .
+ print {$output_handler} "<entry>\n" .
"<title type=\"html\">" . esc_html($co{'title'}) . "</title>\n" .
"<updated>$cd{'iso-8601'}</updated>\n" .
"<author>\n" .
" <name>" . esc_html($co{'author_name'}) . "</name>\n";
if ($co{'author_email'}) {
- print " <email>" . esc_html($co{'author_email'}) . "</email>\n";
+ print {$output_handler} " <email>" . esc_html($co{'author_email'}) . "</email>\n";
}
- print "</author>\n" .
+ print {$output_handler} "</author>\n" .
# use committer for contributor
"<contributor>\n" .
" <name>" . esc_html($co{'committer_name'}) . "</name>\n";
if ($co{'committer_email'}) {
- print " <email>" . esc_html($co{'committer_email'}) . "</email>\n";
+ print {$output_handler} " <email>" . esc_html($co{'committer_email'}) . "</email>\n";
}
- print "</contributor>\n" .
+ print {$output_handler} "</contributor>\n" .
"<published>$cd{'iso-8601'}</published>\n" .
"<link rel=\"alternate\" type=\"text/html\" href=\"$co_url\" />\n" .
"<id>$co_url</id>\n" .
@@ -6709,19 +6726,19 @@ XML
"<div xmlns=\"http://www.w3.org/1999/xhtml\">\n";
}
my $comment = $co{'comment'};
- print "<pre>\n";
+ print {$output_handler} "<pre>\n";
foreach my $line (@$comment) {
$line = esc_html($line);
- print "$line\n";
+ print {$output_handler} "$line\n";
}
- print "</pre><ul>\n";
+ print {$output_handler} "</pre><ul>\n";
foreach my $difftree_line (@difftree) {
my %difftree = parse_difftree_raw_line($difftree_line);
next if !$difftree{'from_id'};
my $file = $difftree{'file'} || $difftree{'to_file'};
- print "<li>" .
+ print {$output_handler} "<li>" .
"[" .
$cgi->a({-href => href(-full=>1, action=>"blobdiff",
hash=>$difftree{'to_id'}, hash_parent=>$difftree{'from_id'},
@@ -6729,26 +6746,26 @@ XML
file_name=>$file, file_parent=>$difftree{'from_file'}),
-title => "diff"}, 'D');
if ($have_blame) {
- print $cgi->a({-href => href(-full=>1, action=>"blame",
+ print {$output_handler} $cgi->a({-href => href(-full=>1, action=>"blame",
file_name=>$file, hash_base=>$commit),
-title => "blame"}, 'B');
}
# if this is not a feed of a file history
if (!defined $file_name || $file_name ne $file) {
- print $cgi->a({-href => href(-full=>1, action=>"history",
+ print {$output_handler} $cgi->a({-href => href(-full=>1, action=>"history",
file_name=>$file, hash=>$commit),
-title => "history"}, 'H');
}
$file = esc_path($file);
- print "] ".
+ print {$output_handler} "] ".
"$file</li>\n";
}
if ($format eq 'rss') {
- print "</ul>]]>\n" .
+ print {$output_handler} "</ul>]]>\n" .
"</content:encoded>\n" .
"</item>\n";
} elsif ($format eq 'atom') {
- print "</ul>\n</div>\n" .
+ print {$output_handler} "</ul>\n</div>\n" .
"</content>\n" .
"</entry>\n";
}
@@ -6756,9 +6773,9 @@ XML
# end of feed
if ($format eq 'rss') {
- print "</channel>\n</rss>\n";
+ print {$output_handler} "</channel>\n</rss>\n";
} elsif ($format eq 'atom') {
- print "</feed>\n";
+ print {$output_handler} "</feed>\n";
}
}
@@ -6772,13 +6789,12 @@ sub git_atom {
sub git_opml {
my @list = git_get_projects_list();
-
- print $cgi->header(
+ print {$output_handler} $cgi->header(
-type => 'text/xml',
-charset => 'utf-8',
-content_disposition => 'inline; filename="opml.xml"');
- print <<XML;
+ print {$output_handler} <<XML;
<?xml version="1.0" encoding="utf-8"?>
<opml version="1.0">
<head>
@@ -6803,9 +6819,9 @@ XML
my $path = esc_html(chop_str($proj{'path'}, 25, 5));
my $rss = href('project' => $proj{'path'}, 'action' => 'rss', -full => 1);
my $html = href('project' => $proj{'path'}, 'action' => 'summary', -full => 1);
- print "<outline type=\"rss\" text=\"$path\" title=\"$path\" xmlUrl=\"$rss\" htmlUrl=\"$html\"/>\n";
+ print {$output_handler} "<outline type=\"rss\" text=\"$path\" title=\"$path\" xmlUrl=\"$rss\" htmlUrl=\"$html\"/>\n";
}
- print <<XML;
+ print {$output_handler} <<XML;
</outline>
</body>
</opml>
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)
2010-01-14 1:23 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle John 'Warthog9' Hawley
@ 2010-01-14 1:23 ` John 'Warthog9' Hawley
2010-01-16 2:48 ` Jakub Narebski
` (2 more replies)
2010-01-16 0:43 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle Jakub Narebski
1 sibling, 3 replies; 60+ messages in thread
From: John 'Warthog9' Hawley @ 2010-01-14 1:23 UTC (permalink / raw)
To: git
This is a very large patch that implements the file based
caching layer that is used on such large sites as kernel.org and
soon git.fedoraproject.org. This provides a simple, and straight
forward caching mechanism that scales dramatically better than
Gitweb by itself.
The caching layer basically buffers the output that Gitweb would
normally return, and saves that output to a cache file on the local
disk. When the file is requested it attempts to gain a shared lock
on the cache file and cat it out to the client. Should an exclusive
lock be on a file (it's being updated) the code has a choice to either
update in the background and go ahead and show the stale page while
update is being performed, or stall the client(s) until the page
is generated.
There are two forms of stalling involved here, background building
and non-background building, both of which are discussed in the
configuration page.
There are still a few known "issues" with respect to this:
- Code needs to be added to be "browser" aware so
that clients like wget that are trying to get a
binary blob don't obtain a "Generating..." page
- There is an intermittent flushing issue that has yet
to be tracked down
Caching is disabled by default with the $cache_enable variable,
setting this to 1 will enable file based caching. It is expected
that this will be extended to include additional types of caching
(like memcached) in the future and should not be exclusively
considered a binary value.
---
gitweb/cache.pm | 283 ++++++++++++++++++++++++++++++++++++++++++++++++++++
gitweb/gitweb.css | 6 +
gitweb/gitweb.perl | 58 ++++++++++-
3 files changed, 344 insertions(+), 3 deletions(-)
create mode 100644 gitweb/cache.pm
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
new file mode 100644
index 0000000..d08bcec
--- /dev/null
+++ b/gitweb/cache.pm
@@ -0,0 +1,283 @@
+# gitweb - simple web interface to track changes in git repositories
+#
+# (C) 2006, John 'Warthog9' Hawley <warthog19@eaglescrag.net>
+#
+# This program is licensed under the GPLv2
+
+#
+# Gitweb caching engine
+#
+
+use File::Path qw(make_path remove_tree);
+use Digest::MD5 qw(md5 md5_hex md5_base64);
+use Fcntl ':flock';
+
+sub cache_fetch {
+ my ($action) = @_;
+ my $cacheTime = 0;
+
+ # Deal with cache being disabled
+ if( $cache_enable == 0 ){
+ $output_handler = *STDOUT;
+ $output_handler_bin = *STDOUT;
+ $actions{$action}->();
+ return;
+ }elsif( $cache_enable == 1 ){
+ #obviously we are using file based caching
+
+ if(! -d $cachedir){
+ print "*** Warning ***: Caching enabled but cache directory does not exsist. ($cachedir)\n";
+ mkdir ("cache", 0665) || die "Cannot create cache dir - you will need to manually create";
+ print "Cache directory created successfully\n";
+ }
+
+ our $full_url = "$my_url?". $ENV{'QUERY_STRING'};
+ our $urlhash = md5_hex($full_url);
+ our $fullhashdir = "$cachedir/". substr( $urlhash, 0, 2) ."/";
+
+ my $numdirs = make_path( $fullhashdir, { mode => 0777, error => \my $mkdirerr, } );
+ if( @$mkdirerr ){
+ my $mkdirerrmsg = "";
+ for my $diag (@$mkdirerr) {
+ my ($file, $message) = %$diag;
+ if($file eq '' ){
+ $mkdirerrmsg .= "general error: $message\n";
+ }else{
+ $mkdirerrmsg .= "problem unlinking $file: $message\n";
+ }
+ }
+ die_error(500, "Could not create cache directory | $mkdirerrmsg");
+ }
+ $fullhashpath = "$fullhashdir/". substr( $urlhash, 2 );
+ $fullhashbinpath = "$fullhashpath.bin";
+ } # done dealing with cache enabled / disabled
+
+ if(! -e "$fullhashpath" ){
+ if(! defined(my $childPid = fork()) ){
+ cacheUpdate($action,0);
+ cacheDisplay($action);
+ } elsif ( $childPid == 0 ){
+ #run the updater
+ cacheUpdate($action,1);
+ }else{
+ cacheWaitForUpdate($action);
+ }
+ }else{
+ #if cache is out dated, update
+ #else displayCache();
+ open(cacheFile, '<', "$fullhashpath");
+ stat(cacheFile);
+ close(cacheFile);
+ $cacheTime = get_loadavg() * 60;
+ if( $cacheTime > $maxCacheTime ){
+ $cacheTime = $maxCacheTime;
+ }
+ if( $cacheTime < $minCacheTime ){
+ $cacheTime = $minCacheTime;
+ }
+ if( (stat(_))[9] < (time - $cacheTime) ){
+ if( ! defined(my $childPid = fork()) ){
+ cacheUpdate($action,0);
+ cacheDisplay($action);
+ } elsif ( $childPid == 0 ){
+ #run the updater
+ #print "Running updater\n";
+ cacheUpdate($action,1);
+ }else{
+ #print "Waiting for update\n";
+ cacheWaitForUpdate($action);
+ }
+ } else {
+ cacheDisplay($action);
+ }
+
+
+ }
+
+ #
+ # If all of the caching failes - lets go ahead and press on without it and fall back to 'default'
+ # non-caching behavior. This is the softest of the failure conditions.
+ #
+ #$actions{$action}->();
+}
+
+sub cacheUpdate {
+ my ($action,$areForked) = @_;
+ my $lockingStatus;
+ my $fileData = "";
+
+ if($backgroundCache){
+ open(cacheFileBG, '>:utf8', "$fullhashpath.bg");
+ my $lockStatBG = flock(cacheFileBG,LOCK_EX|LOCK_NB);
+
+ $lockStatus = $lockStatBG;
+ }else{
+ open(cacheFile, '>:utf8', "$fullhashpath");
+ my $lockStat = flock(cacheFile,LOCK_EX|LOCK_NB);
+
+ $lockStatus = $lockStat;
+ }
+ #print "lock status: $lockStat\n";
+
+
+ if (! $lockStatus ){
+ if ( $areForked ){
+ exit(0);
+ }else{
+ return;
+ }
+ }
+
+ if(
+ $action eq "snapshot"
+ ||
+ $action eq "blob_plain"
+ ){
+ open cacheFileBin, '>', $fullhashbinpath or die_error(500, "Could not open bin dump file");
+ $output_handler_bin = *cacheFileBin;
+ }
+
+ $output_handler = *cacheFile;
+
+ if($backgroundCache){
+ open(cacheFile, '>:utf8', "$fullhashpath");
+ $lockStat = flock(cacheFile,LOCK_EX);
+
+ if (! $lockStat ){
+ if ( $areForked ){
+ exit(0);
+ }else{
+ return;
+ }
+ }
+ }
+
+ $actions{$action}->();
+
+ if(
+ $action eq "snapshot"
+ ||
+ $action eq "blob_plain"
+ ){
+ close(cacheFileBin);
+ }
+
+ flock(cacheFile,LOCK_UN);
+ close(cacheFile);
+
+ if($backgroundCache){
+ flock(cacheFileBG,LOCK_UN);
+ close(cacheFileBG);
+ }
+
+ if ( $areForked ){
+ exit(0);
+ } else {
+ return;
+ }
+}
+
+
+sub cacheWaitForUpdate {
+ my ($action) = @_;
+ my $x = 0;
+ my $max = 10;
+ my $lockStat = 0;
+
+ if( $backgroundCache ){
+ if( -e "$fullhashpath" ){
+ open(cacheFile, '<:utf8', "$fullhashpath");
+ $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
+ stat(cacheFile);
+ close(cacheFile);
+
+ if( $lockStat && ( (stat(_))[9] > (time - $maxCacheLife) ) ){
+ cacheDisplay($action);
+ return;
+ }
+ }
+ }
+
+ if(
+ $action eq "atom"
+ ||
+ $action eq "rss"
+ ||
+ $action eq "opml"
+ ){
+ do {
+ sleep 2 if $x > 0;
+ open(cacheFile, '<:utf8', "$fullhashpath");
+ $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
+ close(cacheFile);
+ $x++;
+ $combinedLockStat = $lockStat;
+ } while ((! $combinedLockStat) && ($x < $max));
+
+ if( $x != $max ){
+ cacheDisplay($action);
+ }
+ return;
+ }
+
+ $| = 1;
+
+ print $::cgi->header(-type=>'text/html', -charset => 'utf-8',
+ -status=> 200, -expires => 'never');
+
+ print <<EOF;
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www/w3.porg/TR/html4/strict.dtd">
+<!-- git web w/caching interface version $version, (C) 2006-2010, John 'Warthog9' Hawley <warthog9\@kernel.org> -->
+<!-- git core binaries version $git_version -->
+<head>
+<meta http-equiv="content-type" content="$content_type; charset=utf-8"/>
+<meta name="generator" content="gitweb/$version git/$git_version"/>
+<meta name="robots" content="index, nofollow"/>
+<meta http-equiv="refresh" content="0"/>
+<title>$title</title>
+</head>
+<body>
+EOF
+
+ print "Generating..";
+ do {
+ print ".";
+ sleep 2 if $x > 0;
+ open(cacheFile, '<:utf8', "$fullhashpath");
+ $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
+ close(cacheFile);
+ $x++;
+ $combinedLockStat = $lockStat;
+ } while ((! $combinedLockStat) && ($x < $max));
+ print <<EOF;
+</body>
+</html>
+EOF
+ return;
+}
+
+sub cacheDisplay {
+ my ($action) = @_;
+ open(cacheFile, '<:utf8', "$fullhashpath");
+ $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
+ if (! $lockStat ){
+ close(cacheFile);
+ cacheWaitForUpdate($action);
+ }
+
+ while( <cacheFile> ){
+ print $_;
+ }
+ if(
+ $action eq "snapshot"
+ ||
+ $action eq "blob_plain"
+ ){
+ open(cacheFileBin, '<', "$fullhashbinpath");
+ binmode STDOUT, ':raw';
+ print <cacheFileBin>;
+ binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi
+ close(cacheFileBin);
+ }
+ close(cacheFile);
+}
diff --git a/gitweb/gitweb.css b/gitweb/gitweb.css
index 50067f2..f809f16 100644
--- a/gitweb/gitweb.css
+++ b/gitweb/gitweb.css
@@ -67,6 +67,12 @@ div.page_path {
border-width: 0px 0px 1px;
}
+div.cachetime {
+ float: left;
+ margin-right: 10px;
+ color: #555555;
+}
+
div.page_footer {
height: 17px;
padding: 4px 8px;
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 8bb323c..ec95bb9 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -230,6 +230,50 @@ our $git_versions_must_match = 1;
# Leave it undefined (or set to 'undef') to turn off load checking.
our $maxload = 300;
+# This enables/disables the caching layer in gitweb. This currently only supports the
+# 'dumb' file based caching layer, primarily used on git.kernel.org. this is reasonably
+# effective but it has the downside of requiring a huge amount of disk space if there
+# are a number of repositories involved. It is not uncommon for git.kernel.org to have
+# on the order of 80G - 120G accumulate over the course of a few months. It is recommended
+# that the cache directory be periodically completely deleted, and this is safe to perform.
+# Suggested mechanism
+# mv $cacheidr $cachedir.flush;mkdir $cachedir;rm -rf $cachedir.flush
+# Value is binary. 0 = disabled (default), 1 = enabled.
+#
+# Values of caching:
+# 1 = 'dumb' file based caching used on git.kernel.org
+our $cache_enable = 0;
+
+# Used to set the minimum cache timeout for the dynamic caching algorithm. Basically
+# if we calculate the cache to be under this number of seconds we set the cache timeout
+# to this minimum.
+# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour
+our $minCacheTime = 20;
+
+# Used to set the maximum cache timeout for the dynamic caching algorithm. Basically
+# if we calculate the cache to exceed this number of seconds we set the cache timeout
+# to this maximum.
+# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour
+our $maxCacheTime = 1200;
+
+# If you need to change the location of the caching directory, override this
+# otherwise this will probably do fine for you
+our $cachedir = 'cache';
+
+# If this is set (to 1) cache will do it's best to always display something instead
+# of making someone wait for the cache to update. This will launch the cacheUpdate
+# into the background and it will lock a <file>.bg file and will only lock the
+# actual cache file when it needs to write into it. In theory this will make
+# gitweb seem more responsive at the price of possibly stale data.
+our $backgroundCache = 1;
+
+# Used to set the maximum cache file life. If a cache files last modify time exceeds
+# this value, it will assume that the data is just too old, and HAS to be regenerated
+# instead of trying to display the existing cache data.
+# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour
+# 18000 = 5 hours
+our $maxCacheLife = 18000;
+
# You define site-wide feature defaults here; override them with
# $GITWEB_CONFIG as necessary.
our %feature = (
@@ -593,6 +637,11 @@ if (defined $maxload && get_loadavg() > $maxload) {
die_error(503, "The load average on the server is too high");
}
+#
+# Includes
+#
+do 'cache.pm';
+
# version of the core git binary
our $git_version = qx("$GIT" --version) =~ m/git version (.*)$/ ? $1 : "unknown";
$number_of_git_cmds++;
@@ -994,7 +1043,7 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&
!$project) {
die_error(400, "Project needed");
}
-$actions{$action}->();
+cache_fetch($action);
exit;
## ======================================================================
@@ -3200,7 +3249,9 @@ sub git_header_html {
# support xhtml+xml but choking when it gets what it asked for.
if (defined $cgi->http('HTTP_ACCEPT') &&
$cgi->http('HTTP_ACCEPT') =~ m/(,|;|\s|^)application\/xhtml\+xml(,|;|\s|$)/ &&
- $cgi->Accept('application/xhtml+xml') != 0) {
+ $cgi->Accept('application/xhtml+xml') != 0
+ &&
+ $cache_enable == 0) {
$content_type = 'application/xhtml+xml';
} else {
$content_type = 'text/html';
@@ -3344,6 +3395,7 @@ sub git_footer_html {
my $feed_class = 'rss_logo';
print {$output_handler} "<div class=\"page_footer\">\n";
+ print {$output_handler} "<div class=\"cachetime\">Cache Last Updated: ". gmtime( time ) ." GMT</div>\n";
if (defined $project) {
my $descr = git_get_project_description($project);
if (defined $descr) {
@@ -3424,7 +3476,7 @@ sub die_error {
my $extra = shift;
# The output handlers for die_error need to be reset to STDOUT
- # so that half the message isn't being output to random and
+ # so that half the message isn't being output to random and
# half to STDOUT as expected. This is mainly for the benefit
# of using git_header_html() and git_footer_html() since those
# internaly use the indirect print handler.
--
1.6.5.2
^ permalink raw reply related [flat|nested] 60+ messages in thread
* Re: [PATCH 0/9] Gitweb caching v5
2010-01-14 1:22 [PATCH 0/9] Gitweb caching v5 John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 1/9] gitweb: Load checking John 'Warthog9' Hawley
@ 2010-01-15 1:40 ` Jakub Narebski
2010-01-15 4:29 ` J.H.
1 sibling, 1 reply; 60+ messages in thread
From: Jakub Narebski @ 2010-01-15 1:40 UTC (permalink / raw)
To: John 'Warthog9' Hawley, John 'Warthog9' Hawley; +Cc: git
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
> Afternoon everyone,
>
> This is the latest incarnation of gitweb w/ caching. This is
> finally at the point where it should probably start either being
> considered for inclusion or mainline, or I need to accept that this
> will never get in and more perminantely fork (as is the case with
> Fedora where this is going in as gitweb-caching as a parrallel rpm
> package).
>
> That said this brings the base up to mainline (again),
Could you tell us which commit is the base of this series (like in
git-request-pull output), i.e. which commit this series is rebased
against?
> it updates a
> number of elements in the caching engine, and this is a much cleaner
> break-out of the tree vs. what I am currently developing against.
Is caching engine part changed since v2?
> v5:
> - Missed a couple of things that were in my local tree, and
> added them back in.
That doesn't tell us much.
> - Split up the die_error and the version matching patch
> - Set version matching to be on by default - otherwise this
> really is code that will never get checked, or at best
> enabled by default by distributions
> - Added a minor code cleanup with respect to $site_header
> that was already in my tree
> - Applied against a more recent git tree vs. 1.6.6-rc2
> - Removed breakout patch for now (did that in v4 actually)
> and will deal with that separately
>
> http://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v5
----
Short comments about patches in this series; I will be sending
detailed comments for each patch individually.
> John 'Warthog9' Hawley (9):
> gitweb: Load checking
Looks good.
> gitweb: change die_error to take "extra" argument for extended die
> information
Commit message could be better (summary should really be shorter), and
I think there is some indent typo, but otherwise looks good.
> gitweb: Add option to force version match
For me it needs to be disabled in gitweb test suite (t/gitweb-lib.sh),
if it is enabled by default. I really like that I can test current
gitweb without need to recompile git.
Also it should have tests that it works as intended (both for matching
and non-matching versions) in t/t9501-gitweb-standalone-http-status.sh
> gitweb: Makefile improvements
Does it differ from my proposal (i.e. gitweb/Makefile doing the work),
based on your idea ("make gitweb" for Makefile and gitweb/Makefile)?
> gitweb: add a get function to compliment print_local_time
> gitweb: add a get function to compliment print_sort_th
Those two looks O.K. from what I seen.
> gitweb: cleanup error message produced by undefined $site_header
Shouldn't there be such protection for other such variables, like
$site_footer and $home_text (and a bit diferent protection against
undefined $projects_list)? By the way, how did you arrived at
undefined $site_header: deafult build configuration leaves it empty,
but defined.
> gitweb: Convert output to using indirect file handle
I have alternate solution, using shorter filehandle name (just $out)
in
git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel
http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel
I would have to think a bit about separate handle for binary files;
I am not sure if it is really required.
> gitweb: File based caching layer (from git.kernel.org)
I am working (time permitting) in spliting this large code drop into
smaller commits, namely:
- href(..., -path_info => 0) (for cache key)
- simple file based caching + tests
- global expire time + tests
- output caching in gitweb (WIP)
- adaptive expiration time (planned)
- tee output / cache write (planned)
- expire time variation from CHI (planned)
- locking for single writer (planned)
- server-side generating info (planned)
- AJAX-y generating info (wishlist)
while ensuring that it pass all existing gitweb tests, and adding new
tests for new features.
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 0/9] Gitweb caching v5
2010-01-15 1:40 ` [PATCH 0/9] Gitweb caching v5 Jakub Narebski
@ 2010-01-15 4:29 ` J.H.
2010-01-15 10:28 ` Jakub Narebski
0 siblings, 1 reply; 60+ messages in thread
From: J.H. @ 2010-01-15 4:29 UTC (permalink / raw)
To: Jakub Narebski; +Cc: John 'Warthog9' Hawley, git
On 01/14/2010 05:40 PM, Jakub Narebski wrote:
> "John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
>
>> Afternoon everyone,
>>
>> This is the latest incarnation of gitweb w/ caching. This is
>> finally at the point where it should probably start either being
>> considered for inclusion or mainline, or I need to accept that this
>> will never get in and more perminantely fork (as is the case with
>> Fedora where this is going in as gitweb-caching as a parrallel rpm
>> package).
>>
>> That said this brings the base up to mainline (again),
>
> Could you tell us which commit is the base of this series (like in
> git-request-pull output), i.e. which commit this series is rebased
> against?
This series was based on
git://git.kernel.org/pub/scm/git/git.git
054d2fa05cf0bc55fe1556c9e87d58d67a144f44
http://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v5
>
>> it updates a
>> number of elements in the caching engine, and this is a much cleaner
>> break-out of the tree vs. what I am currently developing against.
>
> Is caching engine part changed since v2?
Slightly, not dramatically. Only changes were to modify the caching
engine to deal with the change in output mechanism (I.E. my $output ->
print {$fh})
<snip>
>> John 'Warthog9' Hawley (9):
>> gitweb: Load checking
> Looks good.
>
>> gitweb: change die_error to take "extra" argument for extended die
>> information
> Commit message could be better (summary should really be shorter), and
> I think there is some indent typo, but otherwise looks good.
>
>> gitweb: Add option to force version match
> For me it needs to be disabled in gitweb test suite (t/gitweb-lib.sh),
> if it is enabled by default. I really like that I can test current
> gitweb without need to recompile git.
>
> Also it should have tests that it works as intended (both for matching
> and non-matching versions) in t/t9501-gitweb-standalone-http-status.sh
I'll get t9501 cleaned up and make sure that the tests default to
turning it off, and I'll add a test to confirm that this works.
>> gitweb: Makefile improvements
> Does it differ from my proposal (i.e. gitweb/Makefile doing the work),
> based on your idea ("make gitweb" for Makefile and gitweb/Makefile)?
I think this is taken straight from the version you had, I don't think
I've modified it.
>
>> gitweb: add a get function to compliment print_local_time
>> gitweb: add a get function to compliment print_sort_th
> Those two looks O.K. from what I seen.
>
>> gitweb: cleanup error message produced by undefined $site_header
> Shouldn't there be such protection for other such variables, like
> $site_footer and $home_text (and a bit diferent protection against
> undefined $projects_list)? By the way, how did you arrived at
> undefined $site_header: deafult build configuration leaves it empty,
> but defined.
I would have to go back and figure it out, but it's something I hit
years ago and added that check to keep it from spewing all over my logs.
Could easily add it to the others mentioned.
>> gitweb: Convert output to using indirect file handle
> I have alternate solution, using shorter filehandle name (just $out)
> in
>
> git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel
> http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel
>
> I would have to think a bit about separate handle for binary files;
> I am not sure if it is really required.
For caching you have to have it. When your outputing the data back from
the cache you need to switch the output mode for the browser to receive
the data properly. Otherwise the resulting output from the caching
engine is going to be garbage. The caching engine explicitly stores the
binary files separate from the rest of the response.
>> gitweb: File based caching layer (from git.kernel.org)
> I am working (time permitting) in spliting this large code drop into
> smaller commits, namely:
first up, is there a reason not to take the caching layer as it stands
while you work on these? I'm fine with adding test cases for what's
there now if you want, but I guess I'm confused about explicitly wanting
to break these into smaller commits.
> - href(..., -path_info => 0) (for cache key)
You actually *really* want to have the full url vs. just the path_info.
While I accept that this means that you will end up with multiple
copies of data being stored it helps dramatically if you have multiple
sites pointing into the same caching space. If you happen to have two
distinct trees
http://git.public.com/?p=test.git;a=summary
http://git.private.com/?p=test.git;a=summary
That respectively point to:
/group/public/git/test.git
/group/private/git/test.git
you'll end up squashing the cache files needlessly and erroneously as
what's in the cache file will depend on what last site was hit that
generated the file.
> - simple file based caching + tests
> - global expire time + tests
> - output caching in gitweb (WIP)
> - adaptive expiration time (planned)
> - tee output / cache write (planned)
You sadly can't 'tee' the output as this would re-introduce the
stampeding heard problem which is one of the reasons the caching layer
came about in the first place. Suppose you could give one person the
output but make everyone else wait for the cache to finish writing out,
or have the waiting client processes tail the file while it's generated
but those both seem a little excessive vs. just waiting.
> - expire time variation from CHI (planned)
> - locking for single writer (planned)
> - server-side generating info (planned)
> - AJAX-y generating info (wishlist)
If it's helpful I can genuinely devote several more days to this to get
these cleaned up. Which would save you a fair amount of time in
breaking this up.
- John 'Warthog9' Hawley
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 0/9] Gitweb caching v5
2010-01-15 4:29 ` J.H.
@ 2010-01-15 10:28 ` Jakub Narebski
0 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-15 10:28 UTC (permalink / raw)
To: J.H.; +Cc: John 'Warthog9' Hawley, git
On Fri, 15 Jan 2010, J.H. wrote:
> On 01/14/2010 05:40 PM, Jakub Narebski wrote:
>> "John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
>>> it updates a
>>> number of elements in the caching engine, and this is a much cleaner
>>> break-out of the tree vs. what I am currently developing against.
>>
>> Is caching engine part changed since v2?
>
> Slightly, not dramatically. Only changes were to modify the caching
> engine to deal with the change in output mechanism (I.E. my $output ->
> print {$fh})
O.K.
It is still quite large code dump...
>>> John 'Warthog9' Hawley (9):
[...]
>>> gitweb: Add option to force version match
>> For me it needs to be disabled in gitweb test suite (t/gitweb-lib.sh),
>> if it is enabled by default. I really like that I can test current
>> gitweb without need to recompile git.
>>
>> Also it should have tests that it works as intended (both for matching
>> and non-matching versions) in t/t9501-gitweb-standalone-http-status.sh
>
> I'll get t9501 cleaned up and make sure that the tests default to
> turning it off, and I'll add a test to confirm that this works.
I'll send you proposed fixup to squash, so if you want you can simply
wait for a bit.
[...]
>>> gitweb: cleanup error message produced by undefined $site_header
>> Shouldn't there be such protection for other such variables, like
>> $site_footer and $home_text (and a bit diferent protection against
>> undefined $projects_list)? By the way, how did you arrived at
>> undefined $site_header: deafult build configuration leaves it empty,
>> but defined.
>
> I would have to go back and figure it out, but it's something I hit
> years ago and added that check to keep it from spewing all over my logs.
> Could easily add it to the others mentioned.
Please remember that while for $home_text, $site_header and $site_footer
it would be enough to replace
if (-f $file_variable) {
...
}
with
if (defined $file_variable && -f $file_variable) {
...
}
for undefined $projects_list you have to fail (use die_error).
Defensive programming, here protecting against errors in gitweb config,
is a good idea IMHO.
>>> gitweb: Convert output to using indirect file handle
>> I have alternate solution, using shorter filehandle name (just $out)
>> in
>>
>> git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel
>> http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel
>>
>> I would have to think a bit about separate handle for binary files;
>> I am not sure if it is really required.
>
> For caching you have to have it. When your outputing the data back from
> the cache you need to switch the output mode for the browser to receive
> the data properly. Otherwise the resulting output from the caching
> engine is going to be garbage. The caching engine explicitly stores the
> binary files separate from the rest of the response.
I think you don't. Just read from cache file and print to browser in
binary (:raw) mode; IIUC the ':utf8' is needed only when you print from
Perl.
>>> gitweb: File based caching layer (from git.kernel.org)
>> I am working (time permitting) in spliting this large code drop into
>> smaller commits, namely:
>
> first up, is there a reason not to take the caching layer as it stands
> while you work on these? I'm fine with adding test cases for what's
> there now if you want, but I guess I'm confused about explicitly wanting
> to break these into smaller commits.
It's there a reason to take this big code dump while we (me and you) are
working on it? As I see there is no other work on gitweb now, and for the
time being you can run your fork for a bit. Or is it about planned
deployment on git.fedora.org or something? Anyway when there is official
release of git including gitweb output caching, it should be easy to switch
off from caching fork.
The caching code has also a few issues: coding style, Perl style (e.g.
not using $cache_enabled / $caching_enabled as boolean), duplicated code,
unnecessary open/close dance around stat (you can stat file which is not
closed), not using atomic write (I'm not sure how important this issue
is, especially with locking), not being able to replace cache by some
other caching module, etc.
>> - href(..., -path_info => 0) (for cache key)
>
> You actually *really* want to have the full url vs. just the path_info.
Errr... it is not about full url (although I errorneously used relative
URL for human-readable cache key) vs only path_info, but about path_info
(for example http://git.public.com/test.git) and query (for example
http://git.public.com/?p=test.git) URLs which leads to exactly the same
view.
> While I accept that this means that you will end up with multiple
> copies of data being stored it helps dramatically if you have multiple
> sites pointing into the same caching space. If you happen to have two
> distinct trees
>
> http://git.public.com/?p=test.git;a=summary
> http://git.private.com/?p=test.git;a=summary
>
> That respectively point to:
>
> /group/public/git/test.git
> /group/private/git/test.git
>
> you'll end up squashing the cache files needlessly and erroneously as
> what's in the cache file will depend on what last site was hit that
> generated the file.
So the key (the human-readable key, not it's MD5 digest) would be
href(-replay=>1, -full_url=>1, -path_info=>0);
>> - simple file based caching + tests
>> - global expire time + tests
>> - output caching in gitweb (WIP)
>> - adaptive expiration time (planned)
>> - tee output / cache write (planned)
>
> You sadly can't 'tee' the output as this would re-introduce the
> stampeding heard problem which is one of the reasons the caching layer
> came about in the first place. Suppose you could give one person the
> output but make everyone else wait for the cache to finish writing out,
> or have the waiting client processes tail the file while it's generated
> but those both seem a little excessive vs. just waiting.
The "tee" is about serving [one person] the output, while cache is being
written. It is a bit orthogonal issue to the 'stampeding herd' protection.
>> - expire time variation from CHI (planned)
This feature is meant to prevent cache miss stampedes[1]
[1] http://search.cpan.org/~jswartz/CHI-0.33/lib/CHI.pm#expires_variance
>> - locking for single writer (planned)
...or rather locking to avoid 'stampeding herd' problem.
BTW I don't recall if it isn't the solution you used, but if there exists
stale cache entry, the other clients can use stale version instead of
forcing to wait till the writer finishes updating cache.
>> - server-side generating info (planned)
>> - AJAX-y generating info (wishlist)
>
> If it's helpful I can genuinely devote several more days to this to get
> these cleaned up. Which would save you a fair amount of time in
> breaking this up.
I'm in the middle of working on gitweb caching, with the intent of arriving
at (super)set of features that are in your patch series.
See http://repo.or.cz/w/?p=git/jnareb-git.git;a=shortlog;h=refs/heads/gitweb/cache-kernel
(this branch is rebased / amended / rewritten).
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/9] gitweb: Load checking
2010-01-14 1:22 ` [PATCH 1/9] gitweb: Load checking John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information John 'Warthog9' Hawley
@ 2010-01-15 22:30 ` Jakub Narebski
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-15 22:30 UTC (permalink / raw)
To: John 'Warthog9' Hawley; +Cc: git
This one looks good, but while examining other patch in this series
I have noticed rare situation where we would get Perl error with
this patch.
I have added fix-up for this issue, although I guess that better
solution might be not to add any <script> element for git_footer_html
called from die_error.
I'm sorry I haven't noticed this earlier.
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
> From: John 'Warthog9' Hawley <warthog9@kernel.org>
>
> This changes slightly the behavior of gitweb, so that it verifies
> that the box isn't inundated with before attempting to serve gitweb.
> If the box is overloaded, it basically returns a 503 Server Unavailable
> until the load falls below the defined threshold. This helps dramatically
> if you have a box that's I/O bound, reaches a certain load and you
> don't want gitweb, the I/O hog that it is, increasing the pain the
> server is already undergoing.
>
> This behavior is controlled by $maxload configuration variable.
> Default is a load of 300, which for most cases should never be hit.
> Unset it (set it to undefined value, i.e. undef) to turn off checking.
>
> Currently it requires that '/proc/loadavg' file exists, otherwise the
> load check is bypassed (load is taken to be 0). So platforms that do
> not implement '/proc/loadavg' currently cannot use this feature.
> (provisions are included for additional checks to be added by others)
>
While at it check that $action is defined before comparing it in
git_footer_html() subroutine. Until this patch there were no direct
or indirect (via die_error) invocation of git_footer_html() with
$action undefined; each call was after dispatch, which sets $action to
default value if it is undefined.
This would cause Perl error ("Use of uninitialized value in string eq")
if load is too high _and_ gitweb was invoked without action parameter
explicitly set (e.g. for projects list).
> Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>
> Signed-off-by: Jakub Narebski <jnareb@gmail.com>
> ---
> gitweb/README | 7 ++++++-
> gitweb/gitweb.perl | 45 +++++++++++++++++++++++++++++++++++++++++----
> 2 files changed, 47 insertions(+), 5 deletions(-)
>
> diff --git a/gitweb/README b/gitweb/README
> index e34ee79..6c2c8e1 100644
> --- a/gitweb/README
> +++ b/gitweb/README
> @@ -174,7 +174,7 @@ not include variables usually directly set during build):
> Base URL for relative URLs in pages generated by gitweb,
> (e.g. $logo, $favicon, @stylesheets if they are relative URLs),
> needed and used only for URLs with nonempty PATH_INFO via
> - <base href="$base_url>. Usually gitweb sets its value correctly,
> + <base href="$base_url">. Usually gitweb sets its value correctly,
> and there is no need to set this variable, e.g. to $my_uri or "/".
> * $home_link
> Target of the home link on top of all pages (the first part of view
> @@ -228,6 +228,11 @@ not include variables usually directly set during build):
> repositories from launching cross-site scripting (XSS) attacks. Set this
> to true if you don't trust the content of your repositories. The default
> is false.
> + * $maxload
> + Used to set the maximum load that we will still respond to gitweb queries.
> + If server load exceed this value then return "503 Service Unavaliable" error.
> + Server load is taken to be 0 if gitweb cannot determine its value. Set it to
> + undefined value to turn it off. The default is 300.
>
>
> Projects list file format
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 7e477af..0a07d3a 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -221,6 +221,12 @@ our %avatar_size = (
> 'double' => 32
> );
>
> +# Used to set the maximum load that we will still respond to gitweb queries.
> +# If server load exceed this value then return "503 server busy" error.
> +# If gitweb cannot determined server load, it is taken to be 0.
> +# Leave it undefined (or set to 'undef') to turn off load checking.
> +our $maxload = 300;
> +
> # You define site-wide feature defaults here; override them with
> # $GITWEB_CONFIG as necessary.
> our %feature = (
> @@ -551,6 +557,32 @@ if (-e $GITWEB_CONFIG) {
> do $GITWEB_CONFIG_SYSTEM if -e $GITWEB_CONFIG_SYSTEM;
> }
>
> +# Get loadavg of system, to compare against $maxload.
> +# Currently it requires '/proc/loadavg' present to get loadavg;
> +# if it is not present it returns 0, which means no load checking.
> +sub get_loadavg {
> + if( -e '/proc/loadavg' ){
> + open my $fd, '<', '/proc/loadavg'
> + or return 0;
> + my @load = split(/\s+/, scalar <$fd>);
> + close $fd;
> +
> + # The first three columns measure CPU and IO utilization of the last one,
> + # five, and 10 minute periods. The fourth column shows the number of
> + # currently running processes and the total number of processes in the m/n
> + # format. The last column displays the last process ID used.
> + return $load[0] || 0;
> + }
> + # additional checks for load average should go here for things that don't export
> + # /proc/loadavg
> +
> + return 0;
> +}
> +
> +if (defined $maxload && get_loadavg() > $maxload) {
> + die_error(503, "The load average on the server is too high");
> +}
> +
> # version of the core git binary
> our $git_version = qx("$GIT" --version) =~ m/git version (.*)$/ ? $1 : "unknown";
> $number_of_git_cmds++;
> @@ -3354,14 +3386,19 @@ sub git_footer_html {
> # 500: The server isn't configured properly, or
> # an internal error occurred (e.g. failed assertions caused by bugs), or
> # an unknown error occurred (e.g. the git binary died unexpectedly).
> +# 503: The server is currently unavailable (because it is overloaded,
> +# or down for maintenance). Generally, this is a temporary state.
> sub die_error {
> my $status = shift || 500;
> my $error = shift || "Internal server error";
>
> - my %http_responses = (400 => '400 Bad Request',
> - 403 => '403 Forbidden',
> - 404 => '404 Not Found',
> - 500 => '500 Internal Server Error');
> + my %http_responses = (
> + 400 => '400 Bad Request',
> + 403 => '403 Forbidden',
> + 404 => '404 Not Found',
> + 500 => '500 Internal Server Error',
> + 503 => '503 Service Unavailable',
> + );
> git_header_html($http_responses{$status});
> print <<EOF;
> <div class="page_body">
@@ -3354,7 +3354,8 @@ sub git_footer_html {
}
print qq!<script type="text/javascript" src="$javascript"></script>\n!;
- if ($action eq 'blame_incremental') {
+ if (defined $action &&
+ $action eq 'blame_incremental') {
print qq!<script type="text/javascript">\n!.
qq!startBlame("!. href(action=>"blame_data", -replay=>1) .qq!",\n!.
qq! "!. href() .qq!");\n!.
> --
> 1.6.5.2
>
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information
2010-01-14 1:22 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 3/9] gitweb: Add option to force version match John 'Warthog9' Hawley
@ 2010-01-15 22:40 ` Jakub Narebski
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-15 22:40 UTC (permalink / raw)
To: John 'Warthog9' Hawley; +Cc: git
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
Don't you want kernel.org address also for authorship? Also commit
summary can be written in shorter way, see proposal below.
From: "John 'Warthog9' Hawley" <warthog9@kernel.org>
Subject: gitweb: Allow for longer error explanation in die_error()
> This is a small change that just adds a 3rd, optional, parameter to die_error
> that allows for extended error information to be output along with what the
> error was.
Singed-off-by: "John 'Warthog9' Hawley" <warthog9@kernel.org>
> ---
> gitweb/gitweb.perl | 8 +++++++-
> 1 files changed, 7 insertions(+), 1 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 0a07d3a..8298de5 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -3391,6 +3391,7 @@ sub git_footer_html {
> sub die_error {
> my $status = shift || 500;
> my $error = shift || "Internal server error";
> + my $extra = shift;
>
> my %http_responses = (
> 400 => '400 Bad Request',
> @@ -3405,8 +3406,13 @@ sub die_error {
> <br /><br />
> $status - $error
> <br />
> -</div>
> EOF
> + if (defined $extra) {
> + print "<hr />\n" .
> + "$extra\n";
Following gitweb whitespace convention (tab for indent, space for
align), it should be:
+ print "<hr />\n" .
+ "$extra\n";
(6 x space in place of last tab).
> + }
> + print "</div>\n";
> +
> git_footer_html();
> exit;
> }
> --
> 1.6.5.2
>
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/9] gitweb: Add option to force version match
2010-01-14 1:22 ` [PATCH 3/9] gitweb: Add option to force version match John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 4/9] gitweb: Makefile improvements John 'Warthog9' Hawley
@ 2010-01-15 23:36 ` Jakub Narebski
2010-01-24 21:59 ` Petr Baudis
2 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-15 23:36 UTC (permalink / raw)
To: John 'Warthog9' Hawley; +Cc: git
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
> This adds $git_versions_must_match variable, which is set to true,
> checks that we are running on the same version of git that we
> shipped with, and if not throw '500 Internal Server Error' error.
> What is checked is the version of gitweb (embedded in building
> gitweb.cgi), against version of runtime git binary used.
>
> Gitweb can usually run with a mismatched git install. This is more
> here to give an obvious warning as to whats going on vs. silently
> failing.
>
> By default this feature is turned on.
If this feature is turned on, then I would prefer for gitweb tests to
have it explicitly turned off, so I don't need to rebuild git to test
gitweb.
Also it would be nice to have some tests for this new feature.
See patch below (proposed to be squashed with this one).
P.S. t is there where I have noticed the issue with undefined $action
in git_footer_html(), mentioned in reply to patch 1/9.
> Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>
> Signed-off-by: Jakub Narebski <jnareb@gmail.com>
> ---
Here could be information about differences from previous versions of
this patch.
> gitweb/README | 3 +++
> gitweb/gitweb.perl | 23 +++++++++++++++++++++++
> 2 files changed, 26 insertions(+), 0 deletions(-)
>
> diff --git a/gitweb/README b/gitweb/README
> index 6c2c8e1..03151d2 100644
> --- a/gitweb/README
> +++ b/gitweb/README
> @@ -233,6 +233,9 @@ not include variables usually directly set during build):
> If server load exceed this value then return "503 Service Unavaliable" error.
> Server load is taken to be 0 if gitweb cannot determine its value. Set it to
> undefined value to turn it off. The default is 300.
> + * $git_versions_must_match
> + If set, gitweb fails with 500 Internal Server Error if the version of gitweb
> + doesn't match version of git binary. The default is true.
>
>
> Projects list file format
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 8298de5..b41bc33 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -221,6 +221,9 @@ our %avatar_size = (
> 'double' => 32
> );
>
> +# If it is true, exit if gitweb version and git binary version don't match
> +our $git_versions_must_match = 1;
> +
> # Used to set the maximum load that we will still respond to gitweb queries.
> # If server load exceed this value then return "503 server busy" error.
> # If gitweb cannot determined server load, it is taken to be 0.
> @@ -587,6 +590,26 @@ if (defined $maxload && get_loadavg() > $maxload) {
> our $git_version = qx("$GIT" --version) =~ m/git version (.*)$/ ? $1 : "unknown";
> $number_of_git_cmds++;
>
> +# Throw an error if git versions does not match, if $git_versions_must_match is true.
> +if ($git_versions_must_match &&
> + $git_version ne $version) {
> + my $admin_contact =
> + defined $ENV{'SERVER_ADMIN'} ? ", $ENV{'SERVER_ADMIN'}," : '';
> + my $err_msg = <<EOT;
> +<h1 align="center">*** Warning ***</h1>
> +<p>
> +This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,
> +however git version <b>@{[esc_html($git_version)]}</b> was found on server,
> +and administrator requested strict version checking.
> +</p>
> +<p>
> +Please contact the server administrator${admin_contact} to either configure
> +gitweb to allow mismatched versions, or update git or gitweb installation.
> +</p>
> +EOT
> + die_error(500, 'Internal server error', $err_msg);
> +}
> +
> $projects_list ||= $projectroot;
>
> # ======================================================================
diff --git a/t/gitweb-lib.sh b/t/gitweb-lib.sh
index 76d8b7b..d9ffc90
--- a/t/gitweb-lib.sh
+++ b/t/gitweb-lib.sh
@@ -25,6 +25,7 @@ our \$favicon = 'file:///$TEST_DIRECTORY/../gitweb/git-favicon.png';
our \$projects_list = '';
our \$export_ok = '';
our \$strict_export = '';
+our \$git_versions_must_match = 0;
EOF
diff --git a/t/t9501-gitweb-standalone-http-status.sh b/t/t9501-gitweb-standalone-http-status.sh
index 0688a57..721900e 100755
--- a/t/t9501-gitweb-standalone-http-status.sh
+++ b/t/t9501-gitweb-standalone-http-status.sh
@@ -113,5 +113,31 @@ test_expect_success 'snapshots: bad object id' '
'
test_debug 'cat gitweb.output'
+# ======================================================================
+# check $git_versions_must_match feature
+# should be last section, just in case
+cp -f gitweb_config.perl gitweb_config.perl.bak
+echo 'our $git_versions_must_match = 1;' >>gitweb_config.perl
+
+cat <<\EOF >>gitweb_config.perl
+our $version = "current";
+EOF
+test_expect_success 'force version match: no match' '
+ gitweb_run "p=.git" &&
+ grep "500 - Internal Server Error" gitweb.output
+'
+test_debug 'cat gitweb.headers'
+
+cat <<\EOF >>gitweb_config.perl
+# must be kept in sync with code in gitweb/gitweb.perl
+our $version = qx("$GIT" --version) =~ m/git version (.*)$/ ? $1 : "unknown";
+EOF
+test_expect_success 'force version match: match' '
+ gitweb_run "p=.git" &&
+ grep "Status: 200 OK" gitweb.output
+'
+test_debug 'cat gitweb.headers'
+
+mv -f gitweb_config.perl.bak gitweb_config.perl
test_done
> --
> 1.6.5.2
>
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply related [flat|nested] 60+ messages in thread
* Re: [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header
2010-01-14 1:23 ` [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle John 'Warthog9' Hawley
@ 2010-01-15 23:49 ` Jakub Narebski
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-15 23:49 UTC (permalink / raw)
To: John 'Warthog9' Hawley; +Cc: git
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
I have modified commit message a bit
> If $site_header is not defined you get extraneous errors in the web
> logs:
>
> [Wed Jan 13 16:55:42 2010] [error] [client ::1] [Wed Jan 13 16:55:42 2010]
> gitweb.cgi: Use of uninitialized value $site_header in -f at .../gitweb.cgi line 3287.,
> referer: http://git/gitweb.cgi
>
> for example (line wrapped for better readibility). This commit
> ensures that the variable is defined before trying to use it.
Ordinarily build procedure ensures that $site_header is defined (but
empty, therefore false-ish), so this issue might happen only because
of errors in gitweb config file. Nevetheless it is better to code
defensively.
Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>
> ---
> gitweb/gitweb.perl | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 07fdeb5..c4a177d 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -3277,7 +3277,7 @@ EOF
> print "</head>\n" .
> "<body>\n";
>
> - if (-f $site_header) {
> + if ($site_header && -f $site_header) {
It should be, I think
+ if (defined $site_header && -f $site_header) {
although I guess that nobody would name $site_header file "0".
> insert_file($site_header);
> }
>
> --
More important is the fact that it is not the only variable holding
file name, that is not checked that it is defined before use. You
should do the same change also for $site_footer and $home_text, and
die_error(500, "Gitweb misconfigured") or just die_error(500) if
$projects_list is not defined.
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-14 1:23 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org) John 'Warthog9' Hawley
@ 2010-01-16 0:43 ` Jakub Narebski
2010-01-16 0:58 ` Junio C Hamano
2010-01-24 22:14 ` Petr Baudis
1 sibling, 2 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-16 0:43 UTC (permalink / raw)
To: John 'Warthog9' Hawley; +Cc: git
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
> This converts the output handling of gitweb to using an indirect
> file handle. This is in preparation to add the caching layer. This
> is a slight modification to the way I was originally doing it by
> passing the output around. This should be a nop and this shouldn't
> change the behavior of gitweb. This does leave error reporting
> functions (die_error specifically) continuing to output directly
> as I want to garauntee those will report their errors regardless of
> what may be going on with respect to the rest of the output.
Signoff?
Compare with my version of this patch:
http://repo.or.cz/w/git/jnareb-git.git/commitdiff/0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557
in the gitweb/cache-kernel branch:
http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel
My commit message looks like the following:
....
gitweb: Print to explicit filehandle (preparing for caching)
This means replacing
print <something>;
by
print {$out} <something>;
and
binmode STDOUT, <layer>;
by
binmode $out, <layer>;
where $out is global variable set to \*STDOUT at the beginning of
gitweb, but after reading gitweb config. This way it would be simple
to e.g. tie output filehandle or use PerlIO layers to simultaneously
write to standard output and to some specified file (like "tee"
utility does), or redirect output to a scalar, or a file.
die_error (re)sets $out to \*STDOUT; we would (probably) want to treat
errors in a special way, and do not cache them.
The only other differences are reindent of continued lines (if needed),
and sometimes word-wrapping lines which this change made too long.
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
....
> ---
> gitweb/gitweb.perl | 880 ++++++++++++++++++++++++++--------------------------
> 1 files changed, 448 insertions(+), 432 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index c4a177d..8bb323c 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -450,6 +450,13 @@ our %feature = (
> 'default' => [0]},
> );
>
> +# Basic file handler for all of gitweb, there are two of them. The first
> +# is the basic text/html file handler which is used for everything other
> +# then the binary files, that uses a separate file handler though
> +# these are both set to STDOUT for the time being.
> +our $output_handler = *STDOUT;
> +our $output_handler_bin = *STDOUT;
> +
First it is not file handleR, but filehandle.
Second, there is no need for separate filehandle for binary files, if
you do it correctly (i.e. call binmode on filehandle, and not on
STDOUT). When caching is enabled, and 'print {$output_handle} <sth>'
prints to in-memory file (or even directly to cache file) it would do
conversion, so when reading from cache file we can dump it raw, in
binary mode.
Third, wouldn't it be better to use shorter variable name, e.g. $out
or $oh, instead of $output_handle? We would be able to align print(f)
statements without making lines much longer.
Fourth, there is slight difference between
our $out = *STDOUT;
and
out $out = \*STDOUT;
In the former we have global variable, in latter we have indirect
filehandle. CGI::Cache uses the latter form, IIRC.
> @@ -3313,7 +3320,7 @@ EOF
> if ($use_pathinfo) {
> $action .= "/".esc_url($project);
> }
> - print $cgi->startform(-method => "get", -action => $action) .
> + print {$output_handler} $cgi->startform(-method => "get", -action => $action) .
> "<div class=\"search\">\n" .
> (!$use_pathinfo &&
> $cgi->input({-name=>"p", -value=>$project, -type=>"hidden"}) . "\n") .
Here for example after change gitweb source stops being nicely aligned.
OTOH it makes for bigger patch. In my version I did realign.
You can always check for true differences with "diff -w".
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-16 0:43 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle Jakub Narebski
@ 2010-01-16 0:58 ` Junio C Hamano
2010-01-16 1:14 ` Jakub Narebski
2010-01-24 22:14 ` Petr Baudis
1 sibling, 1 reply; 60+ messages in thread
From: Junio C Hamano @ 2010-01-16 0:58 UTC (permalink / raw)
To: Jakub Narebski; +Cc: John 'Warthog9' Hawley, git
Jakub Narebski <jnareb@gmail.com> writes:
> This means replacing
>
> print <something>;
> by
> print {$out} <something>;
Just out of curiosity, how is this different from
print $out <something>;
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-16 0:58 ` Junio C Hamano
@ 2010-01-16 1:14 ` Jakub Narebski
2010-01-16 1:41 ` Junio C Hamano
0 siblings, 1 reply; 60+ messages in thread
From: Jakub Narebski @ 2010-01-16 1:14 UTC (permalink / raw)
To: Junio C Hamano; +Cc: John 'Warthog9' Hawley, git
On Sat, 16 Jan 2010, Junio C Hamano wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
>
> > This means replacing
> >
> > print <something>;
> > by
> > print {$out} <something>;
>
> Just out of curiosity, how is this different from
>
> print $out <something>;
Actually there is no difference. It doesn't matter one way or other in
situations in gitweb.
I have thought however (but I might be mistaken) that "print {$fh} <sth>"
is idiomatic Perl.
'perldoc -f print' says:
Note that if you're storing FILEHANDLES in an array or other expression,
you will have to use a block returning its value instead:
print { $files[$i] } "stuff\n";
print { $OK ? STDOUT : STDERR } "stuff\n";
Also, there is no "," between FILEHANDLE and LIST in "print FILEHANDLE LIST"
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-16 1:14 ` Jakub Narebski
@ 2010-01-16 1:41 ` Junio C Hamano
0 siblings, 0 replies; 60+ messages in thread
From: Junio C Hamano @ 2010-01-16 1:41 UTC (permalink / raw)
To: Jakub Narebski; +Cc: John 'Warthog9' Hawley, git
Jakub Narebski <jnareb@gmail.com> writes:
> I have thought however (but I might be mistaken) that "print {$fh} <sth>"
> is idiomatic Perl.
>
> 'perldoc -f print' says:
> Note that if you're storing FILEHANDLES in an array or other expression,
> you will have to use a block returning its value instead:
Note that "in an array or other expression". I've always thought the
intention of this phrase was "you _could_ help the parser by doing this,
if you have expression more complex than a simple scalar variable
reference".
IOW, I know that {} _can_ be used there, but I haven't seen people write
{$a_single_variable}, especially without a space around the "expression"
(technically, a single variable is an expression), when
print $fh <stuff>
suffices, and I was curious why you chose to use the syntax when it wasn't
necessary. Besides, {$fh} looks so eh... (hesitates to mention a dirty
word ^W^W^Wthe name of a different language, but bleeps it out)...
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)
2010-01-14 1:23 ` [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org) John 'Warthog9' Hawley
@ 2010-01-16 2:48 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
[not found] ` <0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557.1264198194.git.jnareb@gmail.com>
2 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-16 2:48 UTC (permalink / raw)
To: John 'Warthog9' Hawley, John 'Warthog9' Hawley; +Cc: git
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
> This is a very large patch
This is true, and that is why I am woeking on splitting this patch
into series of smaller patches, each adding single feature present in
this megapatch (this code drop)... and cleaning up (and improving) it
while at it. This hopefully would make it easier to review.
> that implements the file based
> caching layer that is used on such large sites as kernel.org and
> soon git.fedoraproject.org. This provides a simple, and straight
> forward caching mechanism that scales dramatically better than
> Gitweb by itself.
Do you have any benchmarks comparing gitweb performace with and
without caching enabled?
>
> The caching layer basically buffers the output that Gitweb would
> normally return, and saves that output to a cache file on the local
> disk. When the file is requested it attempts to gain a shared lock
> on the cache file and cat it out to the client. Should an exclusive
> lock be on a file (it's being updated) the code has a choice to either
> update in the background and go ahead and show the stale page while
> update is being performed, or stall the client(s) until the page
> is generated.
The above paragraph is not very clear to me.
Correct me if I am wrong, but as I understand it the cache
architecture is as following:
* This patch implements output caching, which means that the whole
gitweb response, including HTTP headers, is stored in cache. (This
means that in absence of extra mechanism content-type negotiation
should be disabled when caching is turned on).
* Caching engine used implements simple file based caching layer,
where cached data is stored verbatim in cache file (no serialization
/ hibernating / marshalling of data - better performance, and
possibility of X-Sendfile support). Cache expiration is global
value, i.e. is not stored along cache entry in file. Cache entries
expire based on mtime of file.
* When there exist cache entry for given request, and it is not
expired, gitweb output is served directtly from cache file.
* When there exist cache entry for given request, but it is expired,
one process acquires exclusive (writer) lock on file; the rest of
clients get served stale data.
* When there does not exist cache entry for given request, one process
acquires exclusive (writer) lock on cache file; the rest of clients
wait for cache to be filled.
>
> There are two forms of stalling involved here, background building
> and non-background building, both of which are discussed in the
> configuration page.
I'd like to have at least design decisions put into commit message,
and perhaps also have caching mechanism described in separate section
in gitweb/README.
>
> There are still a few known "issues" with respect to this:
> - Code needs to be added to be "browser" aware so
> that clients like wget that are trying to get a
> binary blob don't obtain a "Generating..." page
This issue should be clearly addressed: when do we serve
"Generating..." page, and when we do not. The issue is not only wget
trying to download binary blob or patchset, or snapshot, but also
binary blob which is image referenced from a blob which is HTML, and
there is issue of web feeds (accessed by feed readers).
> - There is an intermittent flushing issue that has yet
> to be tracked down
Could you tell us more where does this shows (what are the
symptompts)?
BTW if it was split into small separate commits, you could be able to
find bug by bisecting history. Also troubles with finding this bug
might mean that code is not very clean.
>
> Caching is disabled by default with the $cache_enable variable,
> setting this to 1 will enable file based caching. It is expected
> that this will be extended to include additional types of caching
> (like memcached) in the future and should not be exclusively
> considered a binary value.
Not a good idea, IMHO. In my rewrite of this patch there is _boolean_
$caching_enabled variable which controls if (output) caching is
enabled or not, and $cache variable holding instance of cache engine,
which might be used to select different caching that simple file-based
caching.
Signoff?
> ---
> gitweb/cache.pm | 283 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> gitweb/gitweb.css | 6 +
> gitweb/gitweb.perl | 58 ++++++++++-
> 3 files changed, 344 insertions(+), 3 deletions(-)
> create mode 100644 gitweb/cache.pm
Very large patch... but no updates to gitweb/README, no updates to
t/gitweb-lib.sh (I guess that gitweb tests are no longer working).
> diff --git a/gitweb/cache.pm b/gitweb/cache.pm
> new file mode 100644
> index 0000000..d08bcec
> --- /dev/null
> +++ b/gitweb/cache.pm
> @@ -0,0 +1,283 @@
> +# gitweb - simple web interface to track changes in git repositories
> +#
> +# (C) 2006, John 'Warthog9' Hawley <warthog19@eaglescrag.net>
> +#
> +# This program is licensed under the GPLv2
> +
> +#
> +# Gitweb caching engine
> +#
> +
> +use File::Path qw(make_path remove_tree);
Using make_path (you do not use remove_tree, so there is no need for
importing it) instead of older mkdir interface requires File::Path
version 2.0 (which meant that I had to upgrade File::Path). This at
least should be mentioned in the comment, perhaps also in
gitweb/INSTALL.
> +use Digest::MD5 qw(md5 md5_hex md5_base64);
You use only md5_hex; no need to import other functions.
> +use Fcntl ':flock';
> +
> +sub cache_fetch {
> + my ($action) = @_;
> + my $cacheTime = 0;
> +
> + # Deal with cache being disabled
> + if( $cache_enable == 0 ){
Style:
+ if ($cache_enable == 0) {
or better
+ if ($cache_enabled) {
> + $output_handler = *STDOUT;
> + $output_handler_bin = *STDOUT;
There should be no need for that, as $output_handle is set to *STDOUT
(or \*STDOUT) anyway.
> + $actions{$action}->();
> + return;
Anyway I think that the whole block should be _outside_ cache_fetch,
which should be invoked only if caching is enabled. For example in
gitweb.perl:
if ($caching_enabled) {
do $cache_pm;
die $@ if $@;
# ...
cache_fetch($cache, $action);
} else {
$actions{$action}->();
}
> + }elsif( $cache_enable == 1 ){
Style.
> + #obviously we are using file based caching
See my comment about using $cache_enable as enum selecting cache type
(blergh). BTW what's with 'obviously'?
> +
> + if(! -d $cachedir){
Style.
> + print "*** Warning ***: Caching enabled but cache directory does not exsist. ($cachedir)\n";
Why this warning? Is it really necessary?
> + mkdir ("cache", 0665) || die "Cannot create cache dir - you will need to manually create";
> + print "Cache directory created successfully\n";
> + }
> +
> + our $full_url = "$my_url?". $ENV{'QUERY_STRING'};
Wouldn't work if you client uses path_info URL, e.g.
http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel
That's why I use href(-replay=>1, -full_url=>1, -path_info=>0) for
cache key for request (you could use freeze(\%input_params) instead,
where freeze is from Storable module).
> + our $urlhash = md5_hex($full_url);
> + our $fullhashdir = "$cachedir/". substr( $urlhash, 0, 2) ."/";
Is depth 2 enough for cache?
> +
> + my $numdirs = make_path( $fullhashdir, { mode => 0777, error => \my $mkdirerr, } );
> + if( @$mkdirerr ){
> + my $mkdirerrmsg = "";
> + for my $diag (@$mkdirerr) {
> + my ($file, $message) = %$diag;
> + if($file eq '' ){
> + $mkdirerrmsg .= "general error: $message\n";
> + }else{
> + $mkdirerrmsg .= "problem unlinking $file: $message\n";
> + }
> + }
> + die_error(500, "Could not create cache directory | $mkdirerrmsg");
> + }
> + $fullhashpath = "$fullhashdir/". substr( $urlhash, 2 );
> + $fullhashbinpath = "$fullhashpath.bin";
> + } # done dealing with cache enabled / disabled
Note also if dealing with caching enabled / disabled was outside
cache_fetch you would have less nested code.
> +
> + if(! -e "$fullhashpath" ){
> + if(! defined(my $childPid = fork()) ){
Style.
> + cacheUpdate($action,0);
> + cacheDisplay($action);
Why camelCase Java/JavaScript-like convention, quite different from
the C-like naming convention used elsewhere in gitweb?
> + } elsif ( $childPid == 0 ){
> + #run the updater
> + cacheUpdate($action,1);
cacheUpdate($action,0) vs cacheUpdate($action,1) is very cryptic
distinctions. It would be better to use "named parameter" and/or
separate, differently named, [wrapper] functions.
> + }else{
> + cacheWaitForUpdate($action);
> + }
This whole block should probably be in a separate function.
> + }else{
> + #if cache is out dated, update
> + #else displayCache();
> + open(cacheFile, '<', "$fullhashpath");
> + stat(cacheFile);
> + close(cacheFile);
You don't need to open file to stat it.
> + $cacheTime = get_loadavg() * 60;
> + if( $cacheTime > $maxCacheTime ){
> + $cacheTime = $maxCacheTime;
> + }
> + if( $cacheTime < $minCacheTime ){
> + $cacheTime = $minCacheTime;
> + }
This should probably be a separate function (effective cache expiraton
time). Also adaptiveness of caching is not described in commit
message.
> + if( (stat(_))[9] < (time - $cacheTime) ){
> + if( ! defined(my $childPid = fork()) ){
> + cacheUpdate($action,0);
> + cacheDisplay($action);
> + } elsif ( $childPid == 0 ){
> + #run the updater
> + #print "Running updater\n";
Remains of debugging.
> + cacheUpdate($action,1);
> + }else{
> + #print "Waiting for update\n";
> + cacheWaitForUpdate($action);
> + }
Repeated code (I think).
> + } else {
> + cacheDisplay($action);
> + }
> +
> +
> + }
> +
> + #
> + # If all of the caching failes - lets go ahead and press on without it and fall back to 'default'
> + # non-caching behavior. This is the softest of the failure conditions.
> + #
> + #$actions{$action}->();
Why is this commented out?
> +}
> +
> +sub cacheUpdate {
> + my ($action,$areForked) = @_;
> + my $lockingStatus;
> + my $fileData = "";
> +
> + if($backgroundCache){
> + open(cacheFileBG, '>:utf8', "$fullhashpath.bg");
> + my $lockStatBG = flock(cacheFileBG,LOCK_EX|LOCK_NB);
> +
> + $lockStatus = $lockStatBG;
> + }else{
> + open(cacheFile, '>:utf8', "$fullhashpath");
> + my $lockStat = flock(cacheFile,LOCK_EX|LOCK_NB);
> +
> + $lockStatus = $lockStat;
> + }
Almost identical code. Use of global handles instead of indirect
filehandles.
> + #print "lock status: $lockStat\n";
> +
> +
> + if (! $lockStatus ){
> + if ( $areForked ){
> + exit(0);
> + }else{
> + return;
> + }
> + }
This conditional needs explanation (comment), I think.
> +
> + if(
> + $action eq "snapshot"
> + ||
> + $action eq "blob_plain"
This condition should be put in a separate function/
> + ){
> + open cacheFileBin, '>', $fullhashbinpath or die_error(500, "Could not open bin dump file");
> + $output_handler_bin = *cacheFileBin;
> + }
> +
> + $output_handler = *cacheFile;
> +
> + if($backgroundCache){
> + open(cacheFile, '>:utf8', "$fullhashpath");
Why "$fullhashpath" and not simply $fullhashpath?
> + $lockStat = flock(cacheFile,LOCK_EX);
> +
> + if (! $lockStat ){
> + if ( $areForked ){
> + exit(0);
> + }else{
> + return;
> + }
> + }
Repeated code.
> + }
> +
> + $actions{$action}->();
> +
> + if(
> + $action eq "snapshot"
> + ||
> + $action eq "blob_plain"
> + ){
> + close(cacheFileBin);
> + }
> +
> + flock(cacheFile,LOCK_UN);
> + close(cacheFile);
> +
> + if($backgroundCache){
> + flock(cacheFileBG,LOCK_UN);
> + close(cacheFileBG);
> + }
> +
> + if ( $areForked ){
> + exit(0);
> + } else {
> + return;
> + }
> +}
> +
> +
> +sub cacheWaitForUpdate {
> + my ($action) = @_;
> + my $x = 0;
> + my $max = 10;
What is $x, what is $max?
> + my $lockStat = 0;
> +
> + if( $backgroundCache ){
> + if( -e "$fullhashpath" ){
> + open(cacheFile, '<:utf8', "$fullhashpath");
Why opening with :uft8, and not with :raw? I don't think we need to
do the eventual conversion once again...
> + $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
> + stat(cacheFile);
> + close(cacheFile);
> +
> + if( $lockStat && ( (stat(_))[9] > (time - $maxCacheLife) ) ){
> + cacheDisplay($action);
> + return;
> + }
Why do we deal with cache expiration in two places? If it is not
a bug, it should be explained in a comment.
> + }
> + }
> +
> + if(
> + $action eq "atom"
> + ||
> + $action eq "rss"
> + ||
> + $action eq "opml"
> + ){
> + do {
> + sleep 2 if $x > 0;
> + open(cacheFile, '<:utf8', "$fullhashpath");
> + $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
> + close(cacheFile);
> + $x++;
> + $combinedLockStat = $lockStat;
> + } while ((! $combinedLockStat) && ($x < $max));
Why busy wait instead of _blocking_ lock, i.e. waiting on lock for it
to be free? It doesn't look like we _do_ anything in the loop.
Ah, I see that we wait at most 2*$max seconds (where interval of 2
seconds is hardcoded). Is it really necessary?
> +
> + if( $x != $max ){
> + cacheDisplay($action);
> + }
> + return;
> + }
> +
> + $| = 1;
> +
> + print $::cgi->header(-type=>'text/html', -charset => 'utf-8',
> + -status=> 200, -expires => 'never');
> +
> + print <<EOF;
> +<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www/w3.porg/TR/html4/strict.dtd">
> +<!-- git web w/caching interface version $version, (C) 2006-2010, John 'Warthog9' Hawley <warthog9\@kernel.org> -->
> +<!-- git core binaries version $git_version -->
> +<head>
> +<meta http-equiv="content-type" content="$content_type; charset=utf-8"/>
> +<meta name="generator" content="gitweb/$version git/$git_version"/>
> +<meta name="robots" content="index, nofollow"/>
> +<meta http-equiv="refresh" content="0"/>
> +<title>$title</title>
> +</head>
> +<body>
> +EOF
> +
> + print "Generating..";
> + do {
> + print ".";
> + sleep 2 if $x > 0;
> + open(cacheFile, '<:utf8', "$fullhashpath");
> + $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
> + close(cacheFile);
> + $x++;
> + $combinedLockStat = $lockStat;
> + } while ((! $combinedLockStat) && ($x < $max));
This trick of having http-equiv 'refresh' meta with the delay of 0
seconds, but not closing the output and therefore not triggering
redirect should be described in comments, and perhaps also in the
commit message.
> + print <<EOF;
> +</body>
> +</html>
> +EOF
> + return;
> +}
> +
> +sub cacheDisplay {
> + my ($action) = @_;
> + open(cacheFile, '<:utf8', "$fullhashpath");
> + $lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);
> + if (! $lockStat ){
> + close(cacheFile);
> + cacheWaitForUpdate($action);
> + }
> +
> + while( <cacheFile> ){
> + print $_;
> + }
Why not slurp it (local $/ = undef), but write line after line?
> + if(
> + $action eq "snapshot"
> + ||
> + $action eq "blob_plain"
> + ){
> + open(cacheFileBin, '<', "$fullhashbinpath");
> + binmode STDOUT, ':raw';
> + print <cacheFileBin>;
Why not slurp it (local $/ = undef), but write line after line,
implicitly?
> + binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi
> + close(cacheFileBin);
> + }
> + close(cacheFile);
> +}
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 8bb323c..ec95bb9 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -230,6 +230,50 @@ our $git_versions_must_match = 1;
> # Leave it undefined (or set to 'undef') to turn off load checking.
> our $maxload = 300;
>
> +# This enables/disables the caching layer in gitweb. This currently only supports the
> +# 'dumb' file based caching layer, primarily used on git.kernel.org. this is reasonably
> +# effective but it has the downside of requiring a huge amount of disk space if there
> +# are a number of repositories involved. It is not uncommon for git.kernel.org to have
> +# on the order of 80G - 120G accumulate over the course of a few months. It is recommended
> +# that the cache directory be periodically completely deleted, and this is safe to perform.
> +# Suggested mechanism
> +# mv $cacheidr $cachedir.flush;mkdir $cachedir;rm -rf $cachedir.flush
> +# Value is binary. 0 = disabled (default), 1 = enabled.
> +#
> +# Values of caching:
> +# 1 = 'dumb' file based caching used on git.kernel.org
> +our $cache_enable = 0;
> +
> +# Used to set the minimum cache timeout for the dynamic caching algorithm. Basically
> +# if we calculate the cache to be under this number of seconds we set the cache timeout
> +# to this minimum.
> +# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour
> +our $minCacheTime = 20;
> +
> +# Used to set the maximum cache timeout for the dynamic caching algorithm. Basically
> +# if we calculate the cache to exceed this number of seconds we set the cache timeout
> +# to this maximum.
> +# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour
> +our $maxCacheTime = 1200;
> +
> +# If you need to change the location of the caching directory, override this
> +# otherwise this will probably do fine for you
> +our $cachedir = 'cache';
Why not '/tmp/gitweb-cache', or '/var/cache/gitweb'? Perhaps use
TMPDIR / File::Spec->tmpdir() if it is undefined?
Note that this path is relative to the place where we run gitweb from,
which is important for gitweb tests.
> +
> +# If this is set (to 1) cache will do it's best to always display something instead
> +# of making someone wait for the cache to update. This will launch the cacheUpdate
> +# into the background and it will lock a <file>.bg file and will only lock the
> +# actual cache file when it needs to write into it. In theory this will make
> +# gitweb seem more responsive at the price of possibly stale data.
> +our $backgroundCache = 1;
Does it mean that if there exist cache entry for given request, but it
is expired, also the client that created write lock gets stale data
instead of 'Generating...' info, and updates/regenerates cache using
background process?
This comment is not entirely clear for me.
> +
> +# Used to set the maximum cache file life. If a cache files last modify time exceeds
> +# this value, it will assume that the data is just too old, and HAS to be regenerated
> +# instead of trying to display the existing cache data.
> +# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour
> +# 18000 = 5 hours
> +our $maxCacheLife = 18000;
This should also be mentioned in commit message (modifying what I
wrote).
> +
> # You define site-wide feature defaults here; override them with
> # $GITWEB_CONFIG as necessary.
> our %feature = (
> @@ -593,6 +637,11 @@ if (defined $maxload && get_loadavg() > $maxload) {
> die_error(503, "The load average on the server is too high");
> }
>
> +#
> +# Includes
> +#
> +do 'cache.pm';
Should be
+do "$cache_pm";
if you don't use require, where $cache_pm can be overriden in gitweb
config, otherwise gitweb caching tests wouldn't work: they invoke
gitweb from test directory.
> +
> # version of the core git binary
> our $git_version = qx("$GIT" --version) =~ m/git version (.*)$/ ? $1 : "unknown";
> $number_of_git_cmds++;
> @@ -994,7 +1043,7 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&
> !$project) {
> die_error(400, "Project needed");
> }
> -$actions{$action}->();
> +cache_fetch($action);
> exit;
>
As I wrote, I think cache_fetch should be invoked only when caching is
enabled.
> ## ======================================================================
> @@ -3200,7 +3249,9 @@ sub git_header_html {
> # support xhtml+xml but choking when it gets what it asked for.
> if (defined $cgi->http('HTTP_ACCEPT') &&
> $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\s|^)application\/xhtml\+xml(,|;|\s|$)/ &&
> - $cgi->Accept('application/xhtml+xml') != 0) {
> + $cgi->Accept('application/xhtml+xml') != 0
> + &&
> + $cache_enable == 0) {
> $content_type = 'application/xhtml+xml';
> } else {
> $content_type = 'text/html';
O.K.
> @@ -3344,6 +3395,7 @@ sub git_footer_html {
> my $feed_class = 'rss_logo';
>
> print {$output_handler} "<div class=\"page_footer\">\n";
> + print {$output_handler} "<div class=\"cachetime\">Cache Last Updated: ". gmtime( time ) ." GMT</div>\n";
Shouldn't this be conditional on $cache_enabled?
> if (defined $project) {
> my $descr = git_get_project_description($project);
> if (defined $descr) {
BTW. you need, I think, protect timing info and do not show it if
caching is enabled. It doesn't make much sense to show how much time
it took to generate page... when said page could have been retrieved
from cache.
But it might make sense; I am not sure.
> @@ -3424,7 +3476,7 @@ sub die_error {
> my $extra = shift;
>
> # The output handlers for die_error need to be reset to STDOUT
> - # so that half the message isn't being output to random and
> + # so that half the message isn't being output to random and
> # half to STDOUT as expected. This is mainly for the benefit
> # of using git_header_html() and git_footer_html() since those
> # internaly use the indirect print handler.
It looks like spurious change.
> --
> 1.6.5.2
>
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* [RFC PATCH 00/10] gitweb: Simple file based output caching
2010-01-14 1:23 ` [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org) John 'Warthog9' Hawley
2010-01-16 2:48 ` Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1) Jakub Narebski
` (11 more replies)
[not found] ` <0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557.1264198194.git.jnareb@gmail.com>
2 siblings, 12 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
This 10 patches long patch series is intended as proof of concept
for splitting large 'gitweb: File based caching layer (from git.kernel.org)'
mega-patch by John 'Warthog9' Hawley aka J.H., by starting small and
adding features piece by piece.
This patch is meant as replacement for last two patches:
* [PATCH 8/9] gitweb: Convert output to using indirect file handle
Message-ID: <1263432185-21334-9-git-send-email-warthog9@eaglescrag.net>
* [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)
Message-ID: <1263432185-21334-10-git-send-email-warthog9@eaglescrag.net>
in the long patch series by J.H.
* [PATCH 0/9] Gitweb caching v5
http://thread.gmane.org/gmane.comp.version-control.git/136913
Note that this patch series is part of 'gitweb/cache-kernel' branch of
http://repo.or.cz/w/git/jnareb-git.git repository (gitweb link), built
on top of modified patches from 'Gitweb caching v2' series (from
'gitweb-ml-v2' branch of http://git.kernel.org/?p=git/warthog9/gitweb.git
repository). Therefore they might not apply as straight replacements
on top of early parts of 'gitweb-ml-v5' branch.
This is work in progress (showing how I see introducing output caching
to gitweb), it lacks proper documentation (POD for gitweb/cache.pm,
new configuration variables in gitweb/README, perhaps "Gitweb caching"
section in gitweb/README and gitweb/cache.pm mentioned in gitweb/INSTALL),
and commits/patches marked '(WIP)' lacks proper commit message.
Just food for thought...
Table of contents:
~~~~~~~~~~~~~~~~~~
[RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing
for caching)
[RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)
[RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching
[RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration
[RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)
output caching (WIP)
[RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)
[RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)
[RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'
problem (WIP)
[RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for
filling cache (WIP)
[RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
regenerating cache (WIP)
Diffstat:
~~~~~~~~~
gitweb/cache.pm | 566 ++++++++++
gitweb/gitweb.perl | 1923 +++++++++++++++++---------------
t/gitweb-lib.sh | 2 +
t/t9500-gitweb-standalone-no-errors.sh | 13 +
t/t9503-gitweb-caching.sh | 32 +
t/t9503/test_cache_interface.pl | 195 ++++
t/test-lib.sh | 3 +
7 files changed, 1836 insertions(+), 898 deletions(-)
create mode 100644 gitweb/cache.pm
create mode 100755 t/t9503-gitweb-caching.sh
create mode 100755 t/t9503/test_cache_interface.pl
^ permalink raw reply [flat|nested] 60+ messages in thread
* [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching Jakub Narebski
` (10 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
If named boolean option -path_info is passed to href() subroutine, use
its value, instead of querying 'pathinfo' feature to check whether to
generate path_info URL (if generating path_info link is possible at all).
href(-replay=>1, -path_info=>0) is meant to be used to generate a key
for caching gitweb output; alternate solution would be to use freeze()
from Storable (core module) on %input_params hash (or its reference),
e.g.:
$key = freeze \%input_params;
or other serialization technique.
While at it document extra options/flags to href().
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
Actually after discussion with J.H.
http://thread.gmane.org/gmane.comp.version-control.git/136913/focus=137061
I have changed my mind and now I think that good human-readable key
for output caching is href(-replay=>1, -full=>1, -path_info=>0);
The reason behind -path_info=>0 is that we want to have the same cache
entry (the same cache file) regardless of whether we use path_info
URL, or non-path_info URL.
The reason behind -full_path=>1 (from what I understand from
J.H. argument) is that you can have two gitweb deployments sharing the
same cache root, but in which the same project name can point to
different projects (different $projectsroot).
gitweb/gitweb.perl | 7 ++++++-
1 files changed, 6 insertions(+), 1 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 2f6a8e1..cd5073c 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -1004,6 +1004,10 @@ exit;
## ======================================================================
## action links
+# possible values of extra options
+# -full => 0|1 - use absolute/full URL ($my_uri/$my_url as base)
+# -replay => 1 - start from a current view (replay with modifications)
+# -path_info => 0|1 - don't use/use path_info URL (if possible)
sub href {
my %params = @_;
# default is to use -absolute url() i.e. $my_uri
@@ -1020,7 +1024,8 @@ sub href {
}
my $use_pathinfo = gitweb_check_feature('pathinfo');
- if ($use_pathinfo and defined $params{'project'}) {
+ if (defined $params{'project'} &&
+ (exists $params{-path_info} ? $params{-path_info} : $use_pathinfo)) {
# try to put as many parameters as possible in PATH_INFO:
# - project name
# - action
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1) Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration Jakub Narebski
` (9 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
This is first step towards implementing file based output (response)
caching layer that is used on such large sites as kernel.org.
This patch introduces GitwebCaching::SimpleFileCache package, which
follows Cache::Cache / CHI interface, although do not implement it
fully. The intent of following established convention is to be able
in the future to replace our simple file based cache e.g. by one using
memcached.
Like in original patch by John 'Warthog9' Hawley (J.H.) (the one this
commit intends to be incremental step to), the data is stored in the
case as-is, without adding metadata (like expiration date), and
without serialization (which means only scalar data).
To be implemented (from original patch by J.H.):
* cache expiration (based on file stats, current time and global
expiration time); currently elements in cache do not expire
* actually using this cache in gitweb, except error pages
* adaptive cache expiration, based on average system load
* optional locking interface, where only one process can update cache
(using flock)
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Possible extensions (beyond what was in original patch):
* (optionally) show information about cache utilization
* AJAX (JavaScript-based) progress indicator
* JavaScript code to update relative dates in cached output
* make cache size-aware (try to not exceed specified maximum size)
* utilize X-Sendfile header (or equivalent) to show cached data
(optional, as it makes sense only if web server supports sendfile
feature and have it enabled)
* variable expiration feature from CHI, allowing items to expire a bit
earlier than the stated expiration time to prevent cache miss
stampedes (although locking, if available, should take care of
this).
The code of GitwebCaching::SimpleFileCache package in gitweb/cache.pm
was heavily based on file-based cache in Cache::Cache package, i.e.
on Cache::FileCache, Cache::FileBackend and Cache::BaseCache
(including implementing atomic write, something that original patch
lacks).
This patch does not yet enable output caching in gitweb (it doesn't
have all required features yet); on the other hand it includes tests,
currently testing only cache Perl API.
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
Large parts of this code are based _heavily_ on Cache::FileCache
implementation (including Cache::FileBackend and Cache::BaseCache)
from Cache::Cache distribution (which is dual licensed using
(Perl) Artistic License and GNU General Public License, like Perl
itself). In the final version of code it should probably be
cleaned up.
Also although it implements 'compute' interface from CHI (Unified
cache interface), actual CHI code was not used even as reference.
(Nevertheless gitweb/cache.pm is meant to use minimal dependencies,
like gitweb itself, so it won't use Moose / Mouse for OO).
gitweb/cache.pm | 317 +++++++++++++++++++++++++++++++++++++++
t/t9503-gitweb-caching.sh | 32 ++++
t/t9503/test_cache_interface.pl | 77 ++++++++++
t/test-lib.sh | 3 +
4 files changed, 429 insertions(+), 0 deletions(-)
create mode 100644 gitweb/cache.pm
create mode 100755 t/t9503-gitweb-caching.sh
create mode 100755 t/t9503/test_cache_interface.pl
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
new file mode 100644
index 0000000..ea544b0
--- /dev/null
+++ b/gitweb/cache.pm
@@ -0,0 +1,317 @@
+# gitweb - simple web interface to track changes in git repositories
+#
+# (C) 2006, John 'Warthog9' Hawley <warthog19@eaglescrag.net>
+#
+# This program is licensed under the GPLv2
+
+#
+# Gitweb caching engine
+#
+
+{
+# Minimalistic cache that stores data in the filesystem, without serialization
+# and currently without any kind of cache expiration (all keys last forever till
+# they got explicitely removed)
+#
+# It follows Cache::Cache and CHI interface (but does not implement it fully)
+
+package GitwebCache::SimpleFileCache;
+
+use strict;
+use warnings;
+
+use File::Path qw(make_path); # requires version >= 2.0
+use File::Spec;
+use File::Temp;
+use Digest::MD5 qw(md5_hex);
+
+# by default, the cache nests all entries on the filesystem two
+# directories deep
+
+our $DEFAULT_CACHE_DEPTH = 2;
+
+# by default, the root of the cache is located in 'cache'.
+
+our $DEFAULT_CACHE_ROOT = "cache";
+
+# ......................................................................
+# constructor
+
+# The options are set by passing in a reference to a hash containing
+# any of the following keys:
+# * 'namespace'
+# The namespace associated with this cache. This allows easy separation of
+# multiple, distinct caches without worrying about key collision. Defaults
+# to '' (which does not allow for simple implementation of clear() method).
+# * 'cache_root'
+# The location in the filesystem that will hold the root of the cache.
+# Defaults to 'cache', relative to gitweb.cgi directory.
+# * 'cache_depth'
+# The number of subdirectories deep to cache object item. This should be
+# large enough that no cache directory has more than a few hundred objects.
+# Defaults to 2 unless explicitly set.
+sub new {
+ my ($proto, $p_options_hash_ref) = @_;
+
+ my $class = ref($proto) || $proto;
+ my $self = {};
+ $self = bless($self, $class);
+
+ my ($root, $depth, $ns);
+ if (defined $p_options_hash_ref) {
+ $root = $p_options_hash_ref->{'cache_root'};
+ $depth = $p_options_hash_ref->{'cache_depth'};
+ $ns = $p_options_hash_ref->{'namespace'};
+ }
+ $root = $DEFAULT_CACHE_ROOT unless defined($root);
+ $depth = $DEFAULT_CACHE_DEPTH unless defined($depth);
+ $ns = '' unless defined($ns);
+
+ $self->set_root($root);
+ $self->set_depth($depth);
+ $self->set_namespace($ns);
+
+ return $self;
+}
+
+# ......................................................................
+# accessors
+
+sub get_depth {
+ my ($self) = @_;
+
+ return $self->{'_Depth'};
+}
+
+sub set_depth {
+ my ($self, $depth) = @_;
+
+ $self->{'_Depth'} = $depth;
+}
+
+sub get_root {
+ my ($self) = @_;
+
+ return $self->{'_Root'};
+}
+
+
+sub set_root {
+ my ($self, $root) = @_;
+
+ $self->{'_Root'} = $root;
+}
+
+sub get_namespace {
+ my ($self) = @_;
+
+ return $self->{'_Namespace'};
+}
+
+
+sub set_namespace {
+ my ($self, $namespace) = @_;
+
+ $self->{'_Namespace'} = $namespace;
+}
+
+# ----------------------------------------------------------------------
+# (private) utility functions and methods
+
+# Take an human readable key, and create a unique (hashed) key from it
+sub _Build_Hashed_Key {
+ my ($p_key) = @_;
+
+ return md5_hex($p_key);
+}
+
+# Take an human readable key, and return file path
+sub _path_to_key {
+ my ($self, $p_namespace, $p_key) = @_;
+
+ return $self->_path_to_hashed_key($p_namespace,
+ _Build_Hashed_Key($p_key));
+}
+
+# Take hashed key, and return file path
+sub _path_to_hashed_key {
+ my ($self, $p_namespace, $p_hashed_key) = @_;
+
+ return File::Spec->catfile($self->get_root(), $p_namespace,
+ _Split_Word($p_hashed_key, $self->get_depth()));
+}
+
+# Split word into N components, where each component but last is two-letter word
+# e.g. _Split_Word("06b90e786e304a18fdfbd7c7bcc41a6b", 2) == qw(06 b90e786e304a18fdfbd7c7bcc41a6b);
+# _Split_Word("06b90e786e304a18fdfbd7c7bcc41a6b", 3) == qw(06 b9 0e786e304a18fdfbd7c7bcc41a6b);
+sub _Split_Word {
+ my ($p_word, $p_depth) = @_;
+
+ $p_depth--; # now it is number of leading 2-letter components
+ return unpack("(a2)$p_depth a*", $p_word);
+}
+
+sub _Read_File {
+ my ($p_path) = @_;
+
+ -e $p_path
+ or return undef;
+
+ open(my $fh, '<', $p_path)
+ or return undef;
+
+ local $/ = undef;
+ my $data = <$fh>;
+
+ close($fh);
+
+ return $data;
+}
+
+# write a file atomically, assuming that path leading to file exists
+sub _Write_File {
+ my ($p_path, $p_data) = @_;
+
+ my ($volume, $directory, $filename) = File::Spec->splitpath($p_path);
+ if (defined $directory and defined $volume) {
+ $directory = File::Spec->catpath($volume, $directory, '');
+ }
+
+ my $temp = File::Temp->new(DIR => $directory,
+ TEMPLATE => "${filename}_XXXXX",
+ SUFFIX => '.tmp');
+ binmode($temp);
+ print {$temp} $p_data;
+ close($temp);
+
+ rename($temp, $p_path);
+}
+
+# ensures that directory leading to path exists, or dies
+sub _Make_Path {
+ my ($p_path, $p_dir) = @_;
+
+ my ($volume, $directory, $filename) = File::Spec->splitpath($p_path);
+ if (defined $directory and defined $volume) {
+ $directory = File::Spec->catpath($volume, $directory, "");
+ }
+
+ return
+ unless (defined $directory and not -d $directory);
+
+ my $numdirs = make_path($directory,
+ { mode => 0777, error => \my $mkdirerr });
+ if (@$mkdirerr) {
+ my $mkdirerrmsg = "";
+ for my $diag (@$mkdirerr) {
+ my ($file, $message) = %$diag;
+ if ($file eq '' ){
+ $mkdirerrmsg .= "general error: $message\n";
+ } else {
+ $mkdirerrmsg .= "problem unlinking $file: $message\n";
+ }
+ }
+ #die_error(500, "Could not create cache directory | $mkdirerrmsg");
+ }
+}
+
+sub _Remove_File {
+ my ($p_path) = @_;
+
+ if (-f $p_path) {
+ unlink($p_path);
+ }
+}
+
+# _read_data and _write_data methods do deserialization/serialization
+# in original implementation in Cache::Cache distribution
+
+sub _read_data {
+ my ($self, $p_path) = @_;
+
+ return _Read_File($p_path);
+}
+
+sub _write_data {
+ my ($self, $p_path, $p_data) = @_;
+
+ _Make_Path($p_path);
+ _Write_File($p_path, $p_data);
+}
+
+# ----------------------------------------------------------------------
+# worker methods (explicit namespace)
+
+sub restore {
+ my ($self, $p_namespace, $p_key) = @_;
+
+ return $self->_read_data($self->_path_to_key($p_namespace, $p_key));
+}
+
+sub store {
+ my ($self, $p_namespace, $p_key, $p_data) = @_;
+
+ $self->_write_data($self->_path_to_key($p_namespace, $p_key),
+ $p_data);
+}
+
+sub delete_key {
+ my ($self, $p_namespace, $p_key) = @_;
+
+ _Remove_File($self->_path_to_key($p_namespace, $p_key));
+}
+
+sub get_size {
+ my ($self, $p_namespace, $p_key) = @_;
+
+ my $path = $self->_path_to_key($p_namespace, $p_key);
+ if (-e $path) {
+ return -s $path;
+ }
+ return 0;
+}
+
+# ......................................................................
+# interface methods
+
+# Removing and expiring
+
+sub remove {
+ my ($self, $p_key) = @_;
+
+ $self->delete_key($self->get_namespace(), $p_key);
+}
+
+# Getting and setting
+
+sub set {
+ my ($self, $p_key, $p_data) = @_;
+
+ $self->store($self->get_namespace(), $p_key, $p_data);
+}
+
+sub get {
+ my ($self, $p_key) = @_;
+
+ my $data = $self->restore($self->get_namespace(), $p_key)
+ or return undef;
+
+ return $data;
+}
+
+sub compute {
+ my ($self, $p_key, $p_coderef) = @_;
+
+ my $data = $self->get($p_key);
+ if (!defined $data) {
+ $data = $p_coderef->($self, $p_key);
+ $self->set($p_key, $data);
+ }
+
+ return $data;
+}
+
+1;
+} # end of package GitwebCache::SimpleFileCache;
+
+1;
diff --git a/t/t9503-gitweb-caching.sh b/t/t9503-gitweb-caching.sh
new file mode 100755
index 0000000..768080c
--- /dev/null
+++ b/t/t9503-gitweb-caching.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+#
+# Copyright (c) 2010 Jakub Narebski
+#
+
+test_description='caching interface to be used in gitweb'
+#test_description='caching interface used in gitweb, gitweb caching
+#
+#This test checks cache (interface) used in gitweb caching, caching
+#infrastructure and gitweb response (output) caching (the last by
+#running gitweb as CGI script from commandline).'
+
+# for now we are running only cache interface tests
+. ./test-lib.sh
+
+# this test is present in gitweb-lib.sh
+if ! test_have_prereq PERL; then
+ say 'perl not available, skipping test'
+ test_done
+fi
+
+"$PERL_PATH" -MTest::More -e 0 >/dev/null 2>&1 || {
+ say 'perl module Test::More unavailable, skipping test'
+ test_done
+}
+
+# ----------------------------------------------------------------------
+
+test_external 'GitwebCache::* Perl API (in gitweb/cache.pm)' \
+ "$PERL_PATH" "$TEST_DIRECTORY"/t9503/test_cache_interface.pl
+
+test_done
diff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl
new file mode 100755
index 0000000..0b6628b
--- /dev/null
+++ b/t/t9503/test_cache_interface.pl
@@ -0,0 +1,77 @@
+#!/usr/bin/perl
+use lib (split(/:/, $ENV{GITPERLLIB}));
+
+use warnings;
+use strict;
+
+use Test::More;
+
+# test source version; there is no installation target for gitweb
+my $cache_pm = "$ENV{TEST_DIRECTORY}/../gitweb/cache.pm";
+
+unless (-f "$cache_pm") {
+ plan skip_all => "gitweb/cache.pm not found";
+}
+
+# it is currently not a proper Perl module, so we use 'do FILE'
+#ok(eval { do "$cache_pm"; 1 or die $!; }, "loading gitweb/cache.pm");
+my $return = do "$cache_pm";
+ok(!$@, "parse gitweb/cache.pm");
+ok(defined $return, "do gitweb/cache.pm");
+ok($return, "run gitweb/cache.pm");
+# instead of: BEGIN { use_ok('GitwebCache::SimpleFileCache') }
+
+# Test creating a cache
+#
+my $cache = new_ok('GitwebCache::SimpleFileCache',
+ [ { 'cache_root' => 'cache', 'cache_depth' => 2 } ]);
+
+# Test that default values are defined
+#
+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,
+ '$DEFAULT_CACHE_ROOT defined');
+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,
+ '$DEFAULT_CACHE_DEPTH defined');
+
+# Test accessors and default values for cache
+#
+SKIP: {
+ skip 'default values not defined', 3
+ unless ($GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT &&
+ $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH);
+
+ is($cache->get_namespace(), '', "default namespace is ''");
+ is($cache->get_root(), $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,
+ "default cache root is '$GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT'");
+ cmp_ok($cache->get_depth(), '==', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,
+ "default cache depth is $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH");
+}
+
+# Test the getting, setting, and removal of a cached value
+# (Cache::Cache interface)
+#
+my $key = 'Test Key';
+my $value = 'Test Value';
+can_ok($cache, qw(get set remove));
+#ok(!defined($cache->get($key)), 'get before set')
+# or diag("get returned '", $cache->get($key), "' for $key");
+$cache->set($key, $value);
+is($cache->get($key), $value, 'get after set, returns cached value');
+$cache->remove($key);
+ok(!defined($cache->get($key)), 'get after remove, is undefined');
+
+# Test the getting and setting of a cached value
+# (CHI interface)
+#
+my $call_count = 0;
+sub get_value {
+ $call_count++;
+ return $value;
+}
+can_ok($cache, qw(compute));
+is($cache->compute($key, \&get_value), $value, 'compute 1st time (set)');
+is($cache->compute($key, \&get_value), $value, 'compute 2nd time (get)');
+is($cache->compute($key, \&get_value), $value, 'compute 3rd time (get)');
+cmp_ok($call_count, '==', 1, 'get_value() is called once');
+
+done_testing();
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 142f36f..9282d9e 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -429,6 +429,9 @@ test_external () {
# Announce the script to reduce confusion about the
# test output that follows.
say_color "" " run $test_count: $descr ($*)"
+ # Export TEST_DIRECTORY and TRASH_DIRECTORY
+ # to be able to use them in script
+ export TEST_DIRECTORY TRASH_DIRECTORY
# Run command; redirect its stderr to &4 as in
# test_run_, but keep its stdout on our stdout even in
# non-verbose mode.
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1) Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set) output caching (WIP) Jakub Narebski
` (8 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Add stat-based cache expiration to file-based GitwebCache::SimpleFileCache.
Contrary to the way other caching interfaces such as Cache::Cache and CHI
do it, the time cache element expires in is _global_ value associated with
cache instance, and is not local property of cache entry. (Currently cache
entry does not store any metadata associated with entry... which means that
there is no need for serialization / marshalling / freezing and thawing.)
Default expire time is -1, which means never expire.
To check if cache entry is expired, GitwebCache::SimpleFileCache compares
difference between mtime (last modify time) of a cache file and current time
with (global) time to expire. It is done using CHI-compatibile is_valid()
method.
Add test checking that expire time of 0 (expire now) works correctly.
While at it show diagnostic if there were parse errors in gitweb/cache.pm
(TO BE MOVED TO PREVIOUS COMMIT).
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
The main difference from the way J.H. did it in his patch adding output
caching to gitweb is that here I uses imply stat, and not open/stat/close.
Now that cahcing engine supports cache expiration, we can add caching
support to gitweb.
gitweb/cache.pm | 47 ++++++++++++++++++++++++++++++++++++++-
t/t9503/test_cache_interface.pl | 11 ++++++++-
2 files changed, 56 insertions(+), 2 deletions(-)
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
index ea544b0..12a7a78 100644
--- a/gitweb/cache.pm
+++ b/gitweb/cache.pm
@@ -50,6 +50,10 @@ our $DEFAULT_CACHE_ROOT = "cache";
# The number of subdirectories deep to cache object item. This should be
# large enough that no cache directory has more than a few hundred objects.
# Defaults to 2 unless explicitly set.
+# * 'default_expires_in' (Cache::Cache compatibile),
+# 'expires_in' (CHI compatibile) [seconds]
+# The expiration time for objects place in the cache.
+# Defaults to $EXPIRES_NEVER if not explicitly set.
sub new {
my ($proto, $p_options_hash_ref) = @_;
@@ -57,19 +61,24 @@ sub new {
my $self = {};
$self = bless($self, $class);
- my ($root, $depth, $ns);
+ my ($root, $depth, $ns, $expires_in);
if (defined $p_options_hash_ref) {
$root = $p_options_hash_ref->{'cache_root'};
$depth = $p_options_hash_ref->{'cache_depth'};
$ns = $p_options_hash_ref->{'namespace'};
+ $expires_in =
+ $p_options_hash_ref->{'default_expires_in'} ||
+ $p_options_hash_ref->{'expires_in'};
}
$root = $DEFAULT_CACHE_ROOT unless defined($root);
$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);
$ns = '' unless defined($ns);
+ $expires_in = -1 unless defined($expires_in); # <0 means never
$self->set_root($root);
$self->set_depth($depth);
$self->set_namespace($ns);
+ $self->set_expires_in($expires_in);
return $self;
}
@@ -115,6 +124,20 @@ sub set_namespace {
$self->{'_Namespace'} = $namespace;
}
+sub get_expires_in {
+ my ($self) = @_;
+
+ return $self->{'_Expires_In'};
+}
+
+
+sub set_expires_in {
+ my ($self, $expires_in) = @_;
+
+ $self->{'_Expires_In'} = $expires_in;
+}
+
+
# ----------------------------------------------------------------------
# (private) utility functions and methods
@@ -282,6 +305,27 @@ sub remove {
$self->delete_key($self->get_namespace(), $p_key);
}
+# exists in cache and is not expired
+sub is_valid {
+ my ($self, $p_key) = @_;
+
+ # should there be namespace variant of this function?
+ my $path = $self->_path_to_key($self->get_namespace(), $p_key);
+
+ # does file exists in cache?
+ return 0 unless -f $path;
+
+ # expire time can be set to never
+ my $expires_in = $self->get_expires_in();
+ return 1 unless (defined $expires_in && $expires_in >= 0);
+
+ # is file expired?
+ my $mtime = (stat(_))[9];
+ my $now = time();
+
+ return (($now - $mtime) < $expires_in);
+}
+
# Getting and setting
sub set {
@@ -293,6 +337,7 @@ sub set {
sub get {
my ($self, $p_key) = @_;
+ return undef unless $self->is_valid($p_key);
my $data = $self->restore($self->get_namespace(), $p_key)
or return undef;
diff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl
index 0b6628b..3644ca8 100755
--- a/t/t9503/test_cache_interface.pl
+++ b/t/t9503/test_cache_interface.pl
@@ -16,7 +16,8 @@ unless (-f "$cache_pm") {
# it is currently not a proper Perl module, so we use 'do FILE'
#ok(eval { do "$cache_pm"; 1 or die $!; }, "loading gitweb/cache.pm");
my $return = do "$cache_pm";
-ok(!$@, "parse gitweb/cache.pm");
+ok(!$@, "parse gitweb/cache.pm")
+ or diag("parse error:\n", $@);
ok(defined $return, "do gitweb/cache.pm");
ok($return, "run gitweb/cache.pm");
# instead of: BEGIN { use_ok('GitwebCache::SimpleFileCache') }
@@ -74,4 +75,12 @@ is($cache->compute($key, \&get_value), $value, 'compute 2nd time (get)');
is($cache->compute($key, \&get_value), $value, 'compute 3rd time (get)');
cmp_ok($call_count, '==', 1, 'get_value() is called once');
+# Test cache expiration for 'expire now'
+#
+$cache->set_expires_in(0);
+is($cache->get_expires_in(), 0, '"expires in" is set to now (0)');
+$cache->set($key, $value);
+ok(!defined($cache->get($key)), 'cache is expired');
+
+
done_testing();
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set) output caching (WIP)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (2 preceding siblings ...)
2010-01-23 0:27 ` [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP) Jakub Narebski
` (7 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch, as you can see, lack proper commit message: it is work in
progress.
As you can see cache_fetch() subroutine is much, much simpler that the
one in original patch by J.H.:
[PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)
Message-ID: <1263432185-21334-10-git-send-email-warthog9@eaglescrag.net>
http://permalink.gmane.org/gmane.comp.version-control.git/136917
As you can see I have introduced $cache_pm variable, to be able to
test caching in t/t9500-gitweb-standalone-no-errors.sh, but also to be
able to install cache.pm in some other place than along gitweb.cgi.
There would be no such problems if we used 'require GitwebCache' or
somesuch, in place of 'do "cache.pm"' like in original patch by J.H.
But at leats for now I have decided to follow J.H. in this issue.
Instead of using binary (sic!) valued $cache_enable as in J.H. patch,
I use set of two variables: $cache (to be able to select what caching
engine to use, and what features should be enabled), and
$caching_enabled to actually enable/disable cache.
When caching is enabled gitweb do not output timing info (time to
generate page), as it would contain incorrect information if the page
was retrieved from cache; in this place we could put cachetime info
from the original patch by J.H., which means writing
Cache Last Updated: ". gmtime( time )
I have forgot about adding this feature...
ATTENTION !!!: I have run both tests (t9500 to check for errors in
gitweb.perl, and t9503 to test the API), but I haven't actually tested
that *gitweb itself* behaves correctly.
gitweb/cache.pm | 36 ++++++++++++++++++++++++++++
gitweb/gitweb.perl | 40 +++++++++++++++++++++++++++++--
t/gitweb-lib.sh | 2 +
t/t9500-gitweb-standalone-no-errors.sh | 13 ++++++++++
4 files changed, 88 insertions(+), 3 deletions(-)
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
index 12a7a78..3a33158 100644
--- a/gitweb/cache.pm
+++ b/gitweb/cache.pm
@@ -359,4 +359,41 @@ sub compute {
1;
} # end of package GitwebCache::SimpleFileCache;
+# human readable key identifying gitweb output
+sub gitweb_output_key {
+ return href(-replay => 1, -full => 1, -path_info => 0);
+}
+
+sub cache_fetch {
+ my ($cache, $action) = @_;
+
+ my $key = gitweb_output_key();
+ my $data = $cache->get($key);
+
+ if (defined $data) {
+ # print cached data
+ binmode STDOUT, ':raw';
+ print STDOUT $data;
+
+ } else {
+ # calculate data and regenerate data
+ open my $data_fh, '>', \$data
+ or die "Can't open memory file: $!";
+ # matches "binmode STDOUT, ':uft8'" at beginning
+ binmode $data_fh, ':utf8';
+
+ $out = $data_fh || \*STDOUT;
+ $actions{$action}->();
+
+ if (defined $data) {
+ $cache->set($key, $data);
+ binmode STDOUT, ':raw';
+ local $/ = undef;
+ print STDOUT $data;
+ }
+
+ close $data_fh;
+ }
+}
+
1;
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index cd5073c..0394dc8 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -234,6 +234,22 @@ our $gitlinkurl_base = ("++GITWEB_BASE_URL++" =~ m!^(git://.*)$!) ? $1 : '';
# Leave it undefined (or set to 'undef') to turn off load checking.
our $maxload = 300;
+# This enables/disables the caching layer in gitweb. Currently supported
+# is only output (response) caching, similar to the one used on git.kernel.org.
+our $caching_enabled = 0;
+# Set to _initialized_ instance of cache interface implementing (at least)
+# get($key) and set($key, $data) methods (Cache::Cache and CHI interfaces).
+# If unset, GitwebCache::SimpleFileCache would be used, which is 'dumb'
+# (but fast) file based caching layer, currently without any support for
+# cache size limiting. It is therefore recommended that the cache directory
+# be periodically completely deleted; this operation is safe to perform.
+# Suggested mechanism:
+# mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush
+our $cache;
+# Locations of 'cache.pm' file; if it is relative path, it is relative to
+# the directory gitweb is run from
+our $cache_pm = 'cache.pm';
+
# You define site-wide feature defaults here; override them with
# $GITWEB_CONFIG as necessary.
our %feature = (
@@ -998,7 +1014,21 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&
!$project) {
die_error(400, "Project needed");
}
-$actions{$action}->();
+
+if ($caching_enabled) {
+ do $cache_pm;
+ die $@ if $@;
+
+ $cache ||= GitwebCache::SimpleFileCache->new({
+ 'cache_root' => '/tmp/cache',
+ 'cache_depth' => 2,
+ 'expires_in' => 20, # in seconds
+ });
+ cache_fetch($cache, $action);
+} else {
+ $actions{$action}->();
+}
+
exit;
## ======================================================================
@@ -3207,7 +3237,9 @@ sub git_header_html {
# 'application/xhtml+xml', otherwise send it as plain old 'text/html'.
# we have to do this because MSIE sometimes globs '*/*', pretending to
# support xhtml+xml but choking when it gets what it asked for.
- if (defined $cgi->http('HTTP_ACCEPT') &&
+ # Disable content-type negotiation when caching (use mimetype good for all).
+ if (!$caching_enabled &&
+ defined $cgi->http('HTTP_ACCEPT') &&
$cgi->http('HTTP_ACCEPT') =~ m/(,|;|\s|^)application\/xhtml\+xml(,|;|\s|$)/ &&
$cgi->Accept('application/xhtml+xml') != 0) {
$content_type = 'application/xhtml+xml';
@@ -3380,7 +3412,9 @@ sub git_footer_html {
}
print {$out} "</div>\n"; # class="page_footer"
- if (defined $t0 && gitweb_check_feature('timed')) {
+ # timing info doesn't make much sense with output (response) caching
+ if (!$caching_enabled &&
+ defined $t0 && gitweb_check_feature('timed')) {
print {$out} "<div id=\"generating_info\">\n";
print {$out} 'This page took '.
'<span id="generating_time" class="time_span">'.
diff --git a/t/gitweb-lib.sh b/t/gitweb-lib.sh
index d9ffc90..d041083 100755
--- a/t/gitweb-lib.sh
+++ b/t/gitweb-lib.sh
@@ -27,6 +27,8 @@ our \$export_ok = '';
our \$strict_export = '';
our \$git_versions_must_match = 0;
+our \$cache_pm = '$TEST_DIRECTORY/../gitweb/cache.pm';
+
EOF
cat >.git/description <<EOF
diff --git a/t/t9500-gitweb-standalone-no-errors.sh b/t/t9500-gitweb-standalone-no-errors.sh
index 2fc7fdb..0f93962 100755
--- a/t/t9500-gitweb-standalone-no-errors.sh
+++ b/t/t9500-gitweb-standalone-no-errors.sh
@@ -639,4 +639,17 @@ test_expect_success \
gitweb_run "p=.git;a=summary"'
test_debug 'cat gitweb.log'
+# ----------------------------------------------------------------------
+# caching
+
+cat >>gitweb_config.perl <<\EOF
+
+$caching_enabled = 1;
+EOF
+test_expect_success \
+ 'caching enabled' \
+ 'gitweb_run "p=.git;a=summary"'
+test_debug 'cat gitweb.log'
+
+
test_done
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (3 preceding siblings ...)
2010-01-23 0:27 ` [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set) output caching (WIP) Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP) Jakub Narebski
` (6 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch, as you can see, lack proper commit message: it is work in
progress.
As it is quick'n'dirty proof of concept patch, it lacks variables in
gitweb.perl that can be used to control cache behaviour. In original
patch in J.H. those were the following variables: $minCacheTime,
$maxCacheTime.
In the final version we should probably leave $check_load undefined,
or set it to 'sub { return 0; }' instead of requiring that there is
get_loadavg() subroutine in main namespace (see t9503 test change).
It would be alos good idea to leave ->set_expires_in() subroutine
which would set min and max range so that adaptive cache lifetime is
effectively disabled.
gitweb/cache.pm | 82 ++++++++++++++++++++++++++++++++++----
t/t9503/test_cache_interface.pl | 8 +++-
2 files changed, 80 insertions(+), 10 deletions(-)
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
index 3a33158..9692e8d 100644
--- a/gitweb/cache.pm
+++ b/gitweb/cache.pm
@@ -61,24 +61,36 @@ sub new {
my $self = {};
$self = bless($self, $class);
- my ($root, $depth, $ns, $expires_in);
+ my ($root, $depth, $ns);
+ my ($expires_min, $expires_max, $increase_factor, $check_load);
if (defined $p_options_hash_ref) {
$root = $p_options_hash_ref->{'cache_root'};
$depth = $p_options_hash_ref->{'cache_depth'};
$ns = $p_options_hash_ref->{'namespace'};
- $expires_in =
+ $expires_min =
+ $p_options_hash_ref->{'expires_min'} ||
$p_options_hash_ref->{'default_expires_in'} ||
$p_options_hash_ref->{'expires_in'};
+ $expires_max =
+ $p_options_hash_ref->{'expires_max'};
+ $increase_factor = $p_options_hash_ref->{'expires_factor'};
+ $check_load = $p_options_hash_ref->{'check_load'};
}
$root = $DEFAULT_CACHE_ROOT unless defined($root);
$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);
$ns = '' unless defined($ns);
- $expires_in = -1 unless defined($expires_in); # <0 means never
+ $expires_min = 20 unless defined($expires_min);
+ $expires_max = 1200 unless defined($expires_max);
+ $increase_factor = 60 unless defined($increase_factor);
+ $check_load = \&main::get_loadavg unless defined($check_load);
$self->set_root($root);
$self->set_depth($depth);
$self->set_namespace($ns);
- $self->set_expires_in($expires_in);
+ $self->set_expires_min($expires_min);
+ $self->set_expires_max($expires_max);
+ $self->set_increase_factor($increase_factor);
+ $self->set_check_load($check_load);
return $self;
}
@@ -124,19 +136,71 @@ sub set_namespace {
$self->{'_Namespace'} = $namespace;
}
-sub get_expires_in {
+
+sub get_expires_min {
my ($self) = @_;
- return $self->{'_Expires_In'};
+ return $self->{'_Expires_Min'};
+}
+
+sub set_expires_min {
+ my ($self, $expires_min) = @_;
+
+ $self->{'_Expires_Min'} = $expires_min;
}
+sub get_expires_max {
+ my ($self) = @_;
+
+ return $self->{'_Expires_Max'};
+}
+
+sub set_expires_max {
+ my ($self, $expires_max) = @_;
+
+ $self->{'_Expires_Max'} = $expires_max;
+}
+
+sub get_increase_factor {
+ my ($self) = @_;
+
+ return $self->{'_Increase_Factor'};
+}
-sub set_expires_in {
- my ($self, $expires_in) = @_;
+sub set_increase_factor {
+ my ($self, $increase_factor) = @_;
- $self->{'_Expires_In'} = $expires_in;
+ $self->{'_Increase_Factor'} = $increase_factor;
}
+sub get_check_load {
+ my ($self) = @_;
+
+ return $self->{'_Check_Load'};
+}
+
+sub set_check_load {
+ my ($self, $sub) = @_;
+
+ $self->{'_Check_Load'} = $sub;
+}
+
+# ......................................................................
+
+sub get_expires_in {
+ my ($self) = @_;
+ my $expires_in =
+ #$self->get_expires_min() +
+ $self->get_increase_factor() * $self->get_check_load()->();
+
+ if ($expires_in < $self->get_expires_min()) {
+ return $self->get_expires_min();
+ } elsif ($expires_in > $self->get_expires_max()) {
+ return $self->get_expires_max();
+ }
+
+ return $expires_in;
+}
# ----------------------------------------------------------------------
# (private) utility functions and methods
diff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl
index 3644ca8..0870b87 100755
--- a/t/t9503/test_cache_interface.pl
+++ b/t/t9503/test_cache_interface.pl
@@ -6,6 +6,11 @@ use strict;
use Test::More;
+# mockup
+sub get_loadavg {
+ return 0.0;
+}
+
# test source version; there is no installation target for gitweb
my $cache_pm = "$ENV{TEST_DIRECTORY}/../gitweb/cache.pm";
@@ -77,7 +82,8 @@ cmp_ok($call_count, '==', 1, 'get_value() is called once');
# Test cache expiration for 'expire now'
#
-$cache->set_expires_in(0);
+$cache->set_expires_min(0);
+$cache->set_expires_max(0);
is($cache->get_expires_in(), 0, '"expires in" is set to now (0)');
$cache->set($key, $value);
ok(!defined($cache->get($key)), 'cache is expired');
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (4 preceding siblings ...)
2010-01-23 0:27 ` [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP) Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd' problem (WIP) Jakub Narebski
` (5 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch, as you can see, lack proper commit message: it is work in
progress.
The reason behind supporting compute() as interface to cache is that
with this interface it is possible (as it can be seen in the following
patch) to use locking to avoid cache miss stampede (only one process
regenerates cache).
The support for $cache which do not provide '->compute($key, $code)'
method is left just in case we would want to use such (external)
caching engine. We should probably add generic_compute() subroutine
which would use get/set, and provide compute-like interface.
gitweb/cache.pm | 53 +++++++++++++++++++++++++++++++++++++++++++----------
1 files changed, 43 insertions(+), 10 deletions(-)
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
index 9692e8d..8dd4f39 100644
--- a/gitweb/cache.pm
+++ b/gitweb/cache.pm
@@ -432,6 +432,48 @@ sub cache_fetch {
my ($cache, $action) = @_;
my $key = gitweb_output_key();
+ if ($cache->can('compute')) {
+ cache_fetch_compute($cache, $action, $key);
+ } else {
+ cache_fetch_get_set($cache, $action, $key);
+ }
+}
+
+# calculate data to regenerate cache
+sub cache_calculate {
+ my ($action) = @_;
+
+ my $data;
+ open my $data_fh, '>', \$data
+ or die "Can't open memory file: $!";
+ # matches "binmode STDOUT, ':uft8'" at beginning
+ binmode $data_fh, ':utf8';
+
+ $out = $data_fh || \*STDOUT;
+ $actions{$action}->();
+
+ close $data_fh;
+
+ return $data;
+}
+
+# for $cache which can ->compute($key, $code)
+sub cache_fetch_compute {
+ my ($cache, $action, $key) = @_;
+
+ my $data = $cache->compute($key, sub { cache_calculate($action) });
+
+ if (defined $data) {
+ # print cached data
+ binmode STDOUT, ':raw';
+ local $/ = undef;
+ print STDOUT $data;
+ }
+}
+
+# for $cache which can ->get($key) and ->set($key, $data)
+sub cache_fetch_get_set {
+ my ($cache, $action, $key) = @_;
+
my $data = $cache->get($key);
if (defined $data) {
@@ -440,22 +482,13 @@ sub cache_fetch {
print STDOUT $data;
} else {
- # calculate data and regenerate data
- open my $data_fh, '>', \$data
- or die "Can't open memory file: $!";
- # matches "binmode STDOUT, ':uft8'" at beginning
- binmode $data_fh, ':utf8';
-
- $out = $data_fh || \*STDOUT;
- $actions{$action}->();
+ $data = cache_calculate($action);
if (defined $data) {
$cache->set($key, $data);
binmode STDOUT, ':raw';
local $/ = undef;
print STDOUT $data;
}
-
- close $data_fh;
}
}
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd' problem (WIP)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (5 preceding siblings ...)
2010-01-23 0:27 ` [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP) Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for filling cache (WIP) Jakub Narebski
` (4 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch, as you can see, lack proper commit message: it is work in
progress.
As you can see we can almost do the same for the cache which supports
only get/set interface... but for replacing ->retrieve() with ->get()
in ->compute(). This is argument for generic_compute subroutine,
mentioned in previous patch.
Note that we actually can and do test that provided mechanism avoid
cache miss stampede (aka 'stampeding herd') problem... although the
test should probably be run only with --long (this would need update
to t/test-lib.sh to pass GIT_TEST_LONG to external tests in
test_external and the like), as we need to sleep at least one second
to ensure that we would have 'stampeding herd' problem.
gitweb/cache.pm | 27 ++++++++++++++--
t/t9503/test_cache_interface.pl | 66 +++++++++++++++++++++++++++++++++++++++
2 files changed, 90 insertions(+), 3 deletions(-)
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
index 8dd4f39..f514ee9 100644
--- a/gitweb/cache.pm
+++ b/gitweb/cache.pm
@@ -24,6 +24,7 @@ use File::Path qw(make_path); # requires version >= 2.0
use File::Spec;
use File::Temp;
use Digest::MD5 qw(md5_hex);
+use Fcntl qw(:flock);
# by default, the cache nests all entries on the filesystem two
# directories deep
@@ -217,7 +218,7 @@ sub _path_to_key {
my ($self, $p_namespace, $p_key) = @_;
return $self->_path_to_hashed_key($p_namespace,
- _Build_Hashed_Key($p_key));
+ _Build_Hashed_Key($p_key));
}
# Take hashed key, and return file path
@@ -228,6 +229,13 @@ sub _path_to_hashed_key {
_Split_Word($p_hashed_key, $self->get_depth()));
}
+sub _lockfile_to_key {
+ my ($self, $p_namespace, $p_key) = @_;
+
+ return $self->_path_to_hashed_key($p_namespace,
+ _Build_Hashed_Key($p_key)) . '.lock';
+}
+
# Split word into N components, where each component but last is two-letter word
# e.g. _Split_Word("06b90e786e304a18fdfbd7c7bcc41a6b", 2) == qw(06 b90e786e304a18fdfbd7c7bcc41a6b);
# _Split_Word("06b90e786e304a18fdfbd7c7bcc41a6b", 3) == qw(06 b9 0e786e304a18fdfbd7c7bcc41a6b);
@@ -412,17 +420,30 @@ sub compute {
my ($self, $p_key, $p_coderef) = @_;
my $data = $self->get($p_key);
- if (!defined $data) {
+ return $data if defined $data;
+
+ my $lockfile = $self->_lockfile_to_key($self->get_namespace(), $p_key);
+ _Make_Path($lockfile);
+ open my $lock_fh, '+>', $lockfile;
+ # or die "Can't open lockfile '$lockfile': $!";
+ if (my $lock_state = flock($lock_fh, LOCK_EX | LOCK_NB)) {
+ # acquired writers lock
$data = $p_coderef->($self, $p_key);
$self->set($p_key, $data);
+ } else {
+ # get readers lock
+ flock($lock_fh, LOCK_SH);
+ $data = $self->restore($self->get_namespace(), $p_key);
}
-
+ close $lock_fh;
return $data;
}
1;
} # end of package GitwebCache::SimpleFileCache;
+# ======================================================================
+
# human readable key identifying gitweb output
sub gitweb_output_key {
return href(-replay => 1, -full => 1, -path_info => 0);
diff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl
index 0870b87..43b806d 100755
--- a/t/t9503/test_cache_interface.pl
+++ b/t/t9503/test_cache_interface.pl
@@ -65,6 +65,8 @@ $cache->set($key, $value);
is($cache->get($key), $value, 'get after set, returns cached value');
$cache->remove($key);
ok(!defined($cache->get($key)), 'get after remove, is undefined');
+eval { $cache->remove('Not-Existent Key'); };
+ok(!$@, 'remove on non-existent key doesn\'t die');
# Test the getting and setting of a cached value
# (CHI interface)
@@ -80,6 +82,70 @@ is($cache->compute($key, \&get_value), $value, 'compute 2nd time (get)');
is($cache->compute($key, \&get_value), $value, 'compute 3rd time (get)');
cmp_ok($call_count, '==', 1, 'get_value() is called once');
+# Test 'stampeding herd' / cache miss stampede problem
+# (probably should be run only if GIT_TEST_LONG)
+sub get_value_slow {
+ $call_count++;
+ sleep 1;
+ return $value;
+}
+my ($pid, $kid_fh);
+
+$call_count = 0;
+$cache->remove($key);
+$pid = open $kid_fh, '-|';
+SKIP: {
+ skip "cannot fork: $!", 1
+ unless defined $pid;
+
+ my $data = $cache->get($key);
+ if (!defined $data) {
+ $data = get_value_slow();
+ $cache->set($key, $data);
+ }
+
+ if ($pid) {
+ my $child_count = <$kid_fh>;
+ chomp $child_count;
+
+ waitpid $pid, 0;
+ close $kid_fh;
+
+ $call_count += $child_count;
+ } else {
+ print "$call_count\n";
+ exit 0;
+ }
+
+ cmp_ok($call_count, '==', 2, 'parallel get/set: get_value_slow() called twice');
+}
+
+$call_count = 0;
+$cache->remove($key);
+$pid = open $kid_fh, '-|';
+SKIP: {
+ skip "cannot fork: $!", 1
+ unless defined $pid;
+
+ my $data = $cache->compute($key, \&get_value_slow);
+
+ if ($pid) {
+ my $child_count = <$kid_fh>;
+ chomp $child_count;
+
+ waitpid $pid, 0;
+ close $kid_fh;
+
+ $call_count += $child_count;
+ } else {
+ print "$call_count\n";
+ exit 0;
+ }
+
+ cmp_ok($call_count, '==', 1, 'parallel compute: get_value_slow() called once');
+}
+
+
# Test cache expiration for 'expire now'
#
$cache->set_expires_min(0);
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for filling cache (WIP)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (6 preceding siblings ...)
2010-01-23 0:27 ` [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd' problem (WIP) Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating " Jakub Narebski
` (3 subsequent siblings)
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch, as you can see, lack proper commit message: it is work in
progress.
Note that contrary to original patch by J.H. we fork unconditionally;
otherwise the process which do recalculate cache entry would be at
disadvantage compared to processes which failed race to acquire
writers lock. Perhaps this should be made configurable, though.
Also note that currently there is no limit on how stale the data which
is being served while regenerating data can be, i.e. there is no
equivalent for $maxCacheLife. Simplest solution (I guess) would be to
just delete cache entry if it is too stale upfront.
gitweb/cache.pm | 32 +++++++++++++++++++++++++++-----
t/t9503/test_cache_interface.pl | 37 +++++++++++++++++++++++++++++++++++++
2 files changed, 64 insertions(+), 5 deletions(-)
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
index f514ee9..1adf467 100644
--- a/gitweb/cache.pm
+++ b/gitweb/cache.pm
@@ -426,14 +426,36 @@ sub compute {
_Make_Path($lockfile);
open my $lock_fh, '+>', $lockfile;
# or die "Can't open lockfile '$lockfile': $!";
+
+ # try to retrieve stale data
+ $data = $self->restore($self->get_namespace(), $p_key);
+
if (my $lock_state = flock($lock_fh, LOCK_EX | LOCK_NB)) {
# acquired writers lock
- $data = $p_coderef->($self, $p_key);
- $self->set($p_key, $data);
+ my $pid = fork() if $data;
+ if (!defined $pid || $pid) {
+ # parent, or didn't fork
+ $data = $p_coderef->($self, $p_key);
+ $self->set($p_key, $data);
+
+ if ($pid) {
+ # wait for child (which would print) and exit
+ waitpid $pid, 0;
+ exit 0;
+ } else {
+ # there is no child, or was no $data to serve in background
+ ;
+ }
+ } else {
+ # child to serve $data
+ ;
+ }
} else {
- # get readers lock
- flock($lock_fh, LOCK_SH);
- $data = $self->restore($self->get_namespace(), $p_key);
+ if (!defined $data) {
+ # get readers lock if there is no stale data to serve
+ flock($lock_fh, LOCK_SH);
+ $data = $self->restore($self->get_namespace(), $p_key);
+ }
}
close $lock_fh;
return $data;
diff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl
index 43b806d..788e8f2 100755
--- a/t/t9503/test_cache_interface.pl
+++ b/t/t9503/test_cache_interface.pl
@@ -154,5 +154,42 @@ is($cache->get_expires_in(), 0, '"expires in" is set to now (0)');
$cache->set($key, $value);
ok(!defined($cache->get($key)), 'cache is expired');
+# Test that cache returns stale data in existing but expired cache situation
+# (probably should be run only if GIT_TEST_LONG)
+$cache->set_expires_min(0);
+$cache->set_expires_max(0);
+my $stale_value = 'Stale Value';
+my $child_data = '';
+$cache->set($key, $stale_value);
+$call_count = 0;
+$pid = open $kid_fh, '-|';
+SKIP: {
+ skip "cannot fork: $!", 4
+ unless defined $pid;
+
+ my $data = $cache->compute($key, \&get_value_slow);
+
+ if ($pid) {
+ $child_data = <$kid_fh>;
+ chomp $child_data;
+
+ waitpid $pid, 0;
+ close $kid_fh;
+ } else {
+ print "$data\n";
+ exit 0;
+ }
+
+ is($data, $stale_value, 'stale data in parent when expired');
+ is($child_data, $stale_value, 'stale data in child when expired');
+
+ # never expire
+ $cache->set_expires_min(-1);
+ $cache->set_expires_max(-1);
+ is($cache->get($key), $value, 'value got set correctly');
+}
+$cache->set_expires_min(0);
+$cache->set_expires_max(0);
+
done_testing();
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (7 preceding siblings ...)
2010-01-23 0:27 ` [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for filling cache (WIP) Jakub Narebski
@ 2010-01-23 0:27 ` Jakub Narebski
2010-01-24 22:24 ` Petr Baudis
2010-01-23 19:55 ` [RFC PATCH 00/10] gitweb: Simple file based output caching J.H.
` (2 subsequent siblings)
11 siblings, 1 reply; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:27 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch, as you can see, lack proper commit message: it is work in
progress.
Contrary to the patch by J.H. the subroutine that is responsible for
doing "Generating..." progress info / activity indicator is in
gitweb.perl, and not in cache.pm. Also it does not use not declared
nor defined $title variable ;-)
Compared with original patch by J.H. gitweb prints generating info in
more restricted set of situations; the set of actions where gitweb
does not generate activity indicator is larger. We could probably
provide activity indicator also for (possibly) non-HTML output, like
'blob_plain' or 'patches', provided that 'User-Agent' denotes that we
are using web browser.
In the future we would probably want Ajax-y JavaScript-based activity
indicator, without need for busy-wait.
NOTE that I have not tested that it works, only that it should not
contain syntax errors...
gitweb/cache.pm | 32 ++++++++++++++++++++-
gitweb/gitweb.perl | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 104 insertions(+), 3 deletions(-)
diff --git a/gitweb/cache.pm b/gitweb/cache.pm
index 1adf467..2c7be3a 100644
--- a/gitweb/cache.pm
+++ b/gitweb/cache.pm
@@ -63,7 +63,8 @@ sub new {
$self = bless($self, $class);
my ($root, $depth, $ns);
- my ($expires_min, $expires_max, $increase_factor, $check_load);
+ my ($expires_min, $expires_max, $increase_factor,
+ $check_load, $generating_info);
if (defined $p_options_hash_ref) {
$root = $p_options_hash_ref->{'cache_root'};
$depth = $p_options_hash_ref->{'cache_depth'};
@@ -76,6 +77,7 @@ sub new {
$p_options_hash_ref->{'expires_max'};
$increase_factor = $p_options_hash_ref->{'expires_factor'};
$check_load = $p_options_hash_ref->{'check_load'};
+ $generating_info = $p_options_hash_ref->{'generating_info'};
}
$root = $DEFAULT_CACHE_ROOT unless defined($root);
$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);
@@ -92,6 +94,7 @@ sub new {
$self->set_expires_max($expires_max);
$self->set_increase_factor($increase_factor);
$self->set_check_load($check_load);
+ $self->set_generating_info($generating_info);
return $self;
}
@@ -186,6 +189,18 @@ sub set_check_load {
$self->{'_Check_Load'} = $sub;
}
+sub get_generating_info {
+ my ($self) = @_;
+
+ return $self->{'_Generating_Info'};
+}
+
+sub set_generating_info {
+ my ($self, $sub) = @_;
+
+ $self->{'_Generating_Info'} = $sub;
+}
+
# ......................................................................
sub get_expires_in {
@@ -203,6 +218,12 @@ sub get_expires_in {
return $expires_in;
}
+sub generating_info {
+ if (defined $self->get_generating_info()) {
+ $self->get_generating_info()->($self, @_);
+ }
+}
+
# ----------------------------------------------------------------------
# (private) utility functions and methods
@@ -434,6 +455,9 @@ sub compute {
# acquired writers lock
my $pid = fork() if $data;
if (!defined $pid || $pid) {
+ # provide "generating page..." info if there is no stale data to serve
+ $self->generating_info($p_key, $lock_fh)
+ unless ($data);
# parent, or didn't fork
$data = $p_coderef->($self, $p_key);
$self->set($p_key, $data);
@@ -451,8 +475,12 @@ sub compute {
;
}
} else {
+ # some else process is (re)generating cache
if (!defined $data) {
- # get readers lock if there is no stale data to serve
+ # there is no stale data to serve
+ # provide "generating page..." info
+ $self->generating_info($p_key, $lock_fh);
+ # get readers lock
flock($lock_fh, LOCK_SH);
$data = $self->restore($self->get_namespace(), $p_key);
}
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 0394dc8..cb46b8c 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -13,7 +13,7 @@ use CGI qw(:standard :escapeHTML -nosticky);
use CGI::Util qw(unescape);
use CGI::Carp qw(fatalsToBrowser);
use Encode;
-use Fcntl ':mode';
+use Fcntl qw(:mode :flock);
use File::Find qw();
use File::Basename qw(basename);
binmode STDOUT, ':utf8';
@@ -1023,6 +1023,7 @@ if ($caching_enabled) {
'cache_root' => '/tmp/cache',
'cache_depth' => 2,
'expires_in' => 20, # in seconds
+ 'generating_info' => \&git_generating_data_html,
});
cache_fetch($cache, $action);
} else {
@@ -3215,6 +3216,78 @@ sub blob_contenttype {
## ======================================================================
## functions printing HTML: header, footer, error page
+sub git_generating_data_html {
+ my ($cache, $key, $lock_fh) = @_;
+
+ if ($action eq 'atom' || $action eq 'rss' || $action eq 'opml' || # feeds
+ $action eq 'blob_plain' || # unknown mimetype
+ $action eq 'commitdiff_plain' || # text/plain
+ $action eq 'patch' || $action eq 'patches' || # text/plain
+ $action eq 'snapshot') { # binary
+ return;
+ }
+
+ my $title = "[Generating...] $site_name";
+ if (defined $project) {
+ $title .= " - " . to_utf8($project);
+ if (defined $action) {
+ $title .= "/$action";
+ if (defined $file_name) {
+ $title .= " - " . esc_path($file_name);
+ if ($action eq "tree" && $file_name !~ m|/$|) {
+ $title .= "/";
+ }
+ }
+ }
+ }
+
+ my $mod_perl_version = $ENV{'MOD_PERL'} ? " $ENV{'MOD_PERL'}" : '';
+ print STDOUT $cgi->header(-type => 'text/html', -charset => 'utf-8',
+ -status=> '200 OK', -expires => 'now');
+ print STDOUT <<"EOF";
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-US" lang="en-US">
+<!-- git web interface version $version -->
+<!-- git core binaries version $git_version -->
+<head>
+<meta http-equiv="content-type" content="text/html; charset=utf-8" />
+<meta http-equiv="refresh" content="0" />
+<meta name="generator" content="gitweb/$version git/$git_version$mod_perl_version" />
+<meta name="robots" content="noindex, nofollow" />
+<title>$title</title>
+</head>
+<body>
+EOF
+ print STDOUT 'Generating..';
+
+ my $ready;
+ my ($wait, $wait_step, $wait_max); # in seconds
+ $wait = 0;
+ $wait_step = 2;
+ $wait_max = 20;
+ $| = 1; # autoflush
+ do {
+ print STDOUT '.';
+
+ sleep $wait_step if ($wait > 0);
+ $wait += $wait_step;
+
+ $ready = flock($lock_fh, LOCK_SH|LOCK_NB)
+ if $lock_fh;
+
+ } while (!$ready && ($wait < $wait_max));
+
+ print STDOUT <<"EOF";
+
+</body>
+</html>
+EOF
+ #exit 0;
+ return;
+}
+
sub git_header_html {
my $status = shift || "200 OK";
my $expires = shift;
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing for caching)
[not found] ` <0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557.1264198194.git.jnareb@gmail.com>
@ 2010-01-23 0:48 ` Jakub Narebski
2010-02-07 21:32 ` Jakub Narebski
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 0:48 UTC (permalink / raw)
To: git; +Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley
As you can (or rather can't ;-)) see this patch didn't made it into list,
because with 119,993 characters in format-patch patch it probably exceeds
a little bit exceeding 100,000 characters message size limit on VGER.
The problem is that it doesn't make sense to send partial patch... well,
perhaps reindent and breaking of exceedingly long lines should be split
into separate patch...
For now you can view the patch via gitweb
http://repo.or.cz/w/git/jnareb-git.git/commitdiff/0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557
and apply it from
http://repo.or.cz/w/git/jnareb-git.git/patch/0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557
The comment for this message (for this patch) can be seen below.
-- >8 --
On Sat, 23 Jan 2010, Jakub Narebski wrote:
> This means replacing
>
> print <something>;
> by
> print {$out} <something>;
>
> and
>
> binmode STDOUT, <layer>;
> by
> binmode $out, <layer>;
>
> where $out is global variable set to \*STDOUT at the beginning of
> gitweb, but after reading gitweb config. This way it would be simple
> to e.g. tie output filehandle or use PerlIO layers to simultaneously
> write to standard output and to some specified file (like "tee"
> utility does), or redirect output to a scalar, or a file.
>
> die_error (re)sets $out to \*STDOUT; we would (probably) want to treat
> errors in a special way, and do not cache them.
>
>
> The only other differences are reindent of continued lines (if
> needed), and sometimes word-wrapping lines which this change made too
> long.
>
> Signed-off-by: Jakub Narebski <jnareb@gmail.com>
> ---
> This patch is meant as (straight) replacement for the following patch
> by J.H. (John 'Warthog9' Hawley):
> * [PATCH 8/9] gitweb: Convert output to using indirect file handle
> Message-ID: <1263432185-21334-9-git-send-email-warthog9@eaglescrag.net>
> http://permalink.gmane.org/gmane.comp.version-control.git/136915
>
> Actually this patch precedes (was written before) the patch by J.H.
>
> This patch was written _before_ comment from Junio that it would be
> better to simply use
> print $out <something>;
> and do not try to be too clever.
>
>
> Differences from patch by J.H.:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> * Only one output handle, instead of having one output handle for text,
> (':utf8'), and another output handler for binary files (:raw).
>
> I do assume that I can write to handler with appropriate layer:
> :utf8/:raw, and then I can simply read from cache file in :raw
> binmode, as the data is already converted correctly.
>
> * Shorter name for output handle: $out instead of $output_handler
>
> * Set output handler to \*STDOUT (default value in declaration of this
> variable, and also in die_error() subroutine), instead of *STDOUT.
> This way $out is indirect filehandle, instead of using direct
> filehandle which is _global_ to current package (see perlopentut(1)).
>
> * Reindent continuation lines, i.e.
>
> print <line1> .
> <line2>;
>
> got replaced (reindented) with
>
> print {$out} <line1> .
> <line2>;
>
> In some places lines were broken into two, when after indent the
> line got too long.
>
> * Slightly different replacement for printf
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 5/9] gitweb: add a get function to compliment print_local_time
2010-01-14 1:23 ` [PATCH 5/9] gitweb: add a get function to compliment print_local_time John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 6/9] gitweb: add a get function to compliment print_sort_th John 'Warthog9' Hawley
@ 2010-01-23 11:13 ` Jakub Narebski
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-23 11:13 UTC (permalink / raw)
To: John 'Warthog9' Hawley; +Cc: git
"John 'Warthog9' Hawley" <warthog9@eaglescrag.net> writes:
Should authorship be associated with eaglescrag.net or kernel.org?
> This adds a get function for print_local_time so that the basic
> function can be used outside of their straight printing operation.
Signoff?
> ---
> gitweb/gitweb.perl | 11 +++++++++--
> 1 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index b41bc33..d38aad6 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -3537,14 +3537,21 @@ sub git_print_header_div {
> }
>
> sub print_local_time {
> + print get_local_time(@_);
> +}
> +
> +sub get_local_time {
This new function should be rather named 'format_local_time', to
follow gitweb convention of naming subroutines. (And of course commit
message needs to be changed accordingly.)
This comment applies also to the following patch.
> + my $localtime = "";
> my %date = @_;
> if ($date{'hour_local'} < 6) {
> - printf(" (<span class=\"atnight\">%02d:%02d</span> %s)",
> + $localtime .= sprintf(" (<span class=\"atnight\">%02d:%02d</span> %s)",
> $date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});
> } else {
> - printf(" (%02d:%02d %s)",
> + $localtime .= sprintf(" (%02d:%02d %s)",
> $date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});
> }
> +
> + return $localtime
Style (terminate statement with ';'):
+ return $localtime;
> }
>
> # Outputs the author name and date in long form
> --
> 1.6.5.2
>
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 00/10] gitweb: Simple file based output caching
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (8 preceding siblings ...)
2010-01-23 0:27 ` [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating " Jakub Narebski
@ 2010-01-23 19:55 ` J.H.
2010-01-24 13:54 ` [RFC PATCH 11/10] gitweb: Ajax-y "Generating..." page when regenerating cache (WIP) Jakub Narebski
2010-02-06 0:51 ` [RFC PATCH 00/10] gitweb: Simple file based output caching J.H.
11 siblings, 0 replies; 60+ messages in thread
From: J.H. @ 2010-01-23 19:55 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git, John 'Warthog9' Hawley
Just a heads up I wasn't able to get to these yesterday (Friday), but
I'll try and work in looking these over come Monday.
- John 'Warthog9' Hawley
On 01/22/2010 04:27 PM, Jakub Narebski wrote:
> This 10 patches long patch series is intended as proof of concept
> for splitting large 'gitweb: File based caching layer (from git.kernel.org)'
> mega-patch by John 'Warthog9' Hawley aka J.H., by starting small and
> adding features piece by piece.
>
> This patch is meant as replacement for last two patches:
> * [PATCH 8/9] gitweb: Convert output to using indirect file handle
> Message-ID: <1263432185-21334-9-git-send-email-warthog9@eaglescrag.net>
> * [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)
> Message-ID: <1263432185-21334-10-git-send-email-warthog9@eaglescrag.net>
>
> in the long patch series by J.H.
> * [PATCH 0/9] Gitweb caching v5
> http://thread.gmane.org/gmane.comp.version-control.git/136913
>
> Note that this patch series is part of 'gitweb/cache-kernel' branch of
> http://repo.or.cz/w/git/jnareb-git.git repository (gitweb link), built
> on top of modified patches from 'Gitweb caching v2' series (from
> 'gitweb-ml-v2' branch of http://git.kernel.org/?p=git/warthog9/gitweb.git
> repository). Therefore they might not apply as straight replacements
> on top of early parts of 'gitweb-ml-v5' branch.
>
> This is work in progress (showing how I see introducing output caching
> to gitweb), it lacks proper documentation (POD for gitweb/cache.pm,
> new configuration variables in gitweb/README, perhaps "Gitweb caching"
> section in gitweb/README and gitweb/cache.pm mentioned in gitweb/INSTALL),
> and commits/patches marked '(WIP)' lacks proper commit message.
>
> Just food for thought...
>
> Table of contents:
> ~~~~~~~~~~~~~~~~~~
> [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing
> for caching)
> [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)
> [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching
> [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration
> [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)
> output caching (WIP)
> [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)
> [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)
> [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'
> problem (WIP)
> [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for
> filling cache (WIP)
> [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> regenerating cache (WIP)
>
>
> Diffstat:
> ~~~~~~~~~
>
> gitweb/cache.pm | 566 ++++++++++
> gitweb/gitweb.perl | 1923 +++++++++++++++++---------------
> t/gitweb-lib.sh | 2 +
> t/t9500-gitweb-standalone-no-errors.sh | 13 +
> t/t9503-gitweb-caching.sh | 32 +
> t/t9503/test_cache_interface.pl | 195 ++++
> t/test-lib.sh | 3 +
> 7 files changed, 1836 insertions(+), 898 deletions(-)
> create mode 100644 gitweb/cache.pm
> create mode 100755 t/t9503-gitweb-caching.sh
> create mode 100755 t/t9503/test_cache_interface.pl
^ permalink raw reply [flat|nested] 60+ messages in thread
* [RFC PATCH 11/10] gitweb: Ajax-y "Generating..." page when regenerating cache (WIP)
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (9 preceding siblings ...)
2010-01-23 19:55 ` [RFC PATCH 00/10] gitweb: Simple file based output caching J.H.
@ 2010-01-24 13:54 ` Jakub Narebski
2010-02-06 0:51 ` [RFC PATCH 00/10] gitweb: Simple file based output caching J.H.
11 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-24 13:54 UTC (permalink / raw)
To: git
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley,
Jakub Narebski
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This patch, as you can see, lack proper commit message: it is work in
progress (and very much proof of concept patch).
This is Ajax-y JavaScript-based activity indicator. It is very
bare-bones, without any error checking or fallback in the case we
misdetected and JavaScript is turned off. Also currently it would
work only for subsequent links from gitweb, with 'js=1' added.
We need to mark Ajax request to _not_ create "Generating..." page.
I have used there custon 'X-Ajax' HTTP header; we could use
'Pragma: wait', or ejen additional query parameter for that.
gitweb/gitweb.perl | 69 +++++++++++++++++++++++++++++++++++++--------------
1 files changed, 50 insertions(+), 19 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index cb46b8c..d422a4b 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3219,6 +3219,9 @@ sub blob_contenttype {
sub git_generating_data_html {
my ($cache, $key, $lock_fh) = @_;
+ if ($cgi->http('X-Ajax')) {
+ return;
+ }
if ($action eq 'atom' || $action eq 'rss' || $action eq 'opml' || # feeds
$action eq 'blob_plain' || # unknown mimetype
$action eq 'commitdiff_plain' || # text/plain
@@ -3241,6 +3244,7 @@ sub git_generating_data_html {
}
}
+ my $use_javascript = $cgi->param('js');
my $mod_perl_version = $ENV{'MOD_PERL'} ? " $ENV{'MOD_PERL'}" : '';
print STDOUT $cgi->header(-type => 'text/html', -charset => 'utf-8',
-status=> '200 OK', -expires => 'now');
@@ -3253,31 +3257,58 @@ sub git_generating_data_html {
<!-- git core binaries version $git_version -->
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
-<meta http-equiv="refresh" content="0" />
+EOF
+ if (!$use_javascript) {
+ print STDOUT qq!<meta http-equiv="refresh" content="0" />\n!;
+ }
+ print STDOUT <<"EOF";
<meta name="generator" content="gitweb/$version git/$git_version$mod_perl_version" />
<meta name="robots" content="noindex, nofollow" />
<title>$title</title>
</head>
<body>
EOF
- print STDOUT 'Generating..';
-
- my $ready;
- my ($wait, $wait_step, $wait_max); # in seconds
- $wait = 0;
- $wait_step = 2;
- $wait_max = 20;
- $| = 1; # autoflush
- do {
- print STDOUT '.';
-
- sleep $wait_step if ($wait > 0);
- $wait += $wait_step;
-
- $ready = flock($lock_fh, LOCK_SH|LOCK_NB)
- if $lock_fh;
-
- } while (!$ready && ($wait < $wait_max));
+ if ($use_javascript) {
+ my $url_self = href(-replay=>1);
+ print STDOUT <<"EOF";
+Generating...<br />
+<script type="text/javascript" src="$javascript"></script>
+<script type="text/javascript">
+window.onload = function () {
+ xhr = createRequestObject();
+ xhr.onreadystatechange = function () {
+ if (xhr.readyState === 4 && xhr.status === 200) {
+ // intentionally overwrite document
+ document.write(xhr.responseText);
+ }
+ }
+ xhr.open('GET', $url_self);
+ // mark request to not show "Generating..." page
+ xhr.setRequestHeader('X-Ajax', 'Yes');
+ xhr.send(null);
+}
+</script>
+EOF
+ } else {
+ print STDOUT 'Generating..';
+
+ my $ready;
+ my ($wait, $wait_step, $wait_max); # in seconds
+ $wait = 0;
+ $wait_step = 2;
+ $wait_max = 20;
+ $| = 1; # autoflush
+ do {
+ print STDOUT '.';
+
+ sleep $wait_step if ($wait > 0);
+ $wait += $wait_step;
+
+ $ready = flock($lock_fh, LOCK_SH|LOCK_NB)
+ if $lock_fh;
+
+ } while (!$ready && ($wait < $wait_max));
+ }
print STDOUT <<"EOF";
--
1.6.6.1
^ permalink raw reply related [flat|nested] 60+ messages in thread
* Re: [PATCH 3/9] gitweb: Add option to force version match
2010-01-14 1:22 ` [PATCH 3/9] gitweb: Add option to force version match John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 4/9] gitweb: Makefile improvements John 'Warthog9' Hawley
2010-01-15 23:36 ` [PATCH 3/9] gitweb: Add option to force version match Jakub Narebski
@ 2010-01-24 21:59 ` Petr Baudis
2010-01-24 23:17 ` Jakub Narebski
2 siblings, 1 reply; 60+ messages in thread
From: Petr Baudis @ 2010-01-24 21:59 UTC (permalink / raw)
To: John 'Warthog9' Hawley; +Cc: git
Hi!
On Wed, Jan 13, 2010 at 05:22:59PM -0800, John 'Warthog9' Hawley wrote:
> +# Throw an error if git versions does not match, if $git_versions_must_match is true.
> +if ($git_versions_must_match &&
> + $git_version ne $version) {
> + my $admin_contact =
> + defined $ENV{'SERVER_ADMIN'} ? ", $ENV{'SERVER_ADMIN'}," : '';
> + my $err_msg = <<EOT;
> +<h1 align="center">*** Warning ***</h1>
> +<p>
> +This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,
> +however git version <b>@{[esc_html($git_version)]}</b> was found on server,
> +and administrator requested strict version checking.
> +</p>
> +<p>
> +Please contact the server administrator${admin_contact} to either configure
> +gitweb to allow mismatched versions, or update git or gitweb installation.
> +</p>
> +EOT
> + die_error(500, 'Internal server error', $err_msg);
> +}
> +
I get very unhappy when *I* am the server administrator and read
messages like this. ;-) Could you please mention the
\$git_versions_must_match in the text?
Petr "Pasky" Baudis
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-16 0:43 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle Jakub Narebski
2010-01-16 0:58 ` Junio C Hamano
@ 2010-01-24 22:14 ` Petr Baudis
2010-01-25 1:47 ` Jakub Narebski
1 sibling, 1 reply; 60+ messages in thread
From: Petr Baudis @ 2010-01-24 22:14 UTC (permalink / raw)
To: Jakub Narebski; +Cc: John 'Warthog9' Hawley, git
On Fri, Jan 15, 2010 at 04:43:32PM -0800, Jakub Narebski wrote:
> Third, wouldn't it be better to use shorter variable name, e.g. $out
> or $oh, instead of $output_handle? We would be able to align print(f)
> statements without making lines much longer.
I totally agree, I was going to comment on this too. Please use
something very short, I think even the three characters might be pushing
it a bit. Having long idiom will make it real pain to both read and
write gitweb code.
It would still be nice if we could have some way to avoid this hack
entirely. Couldn't we layer PerlIO::via over STDOUT and continue to use
argument-less print as before?
I think we (well, in practice, "you" nowadays :) should be less
perfectionist when accepting gitweb contributions, so if the answer is
"it's too hard for me to do right now", that's fine I guess; I know I
can't contribute the code currently. :(
Petr "Pasky" Baudis
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-23 0:27 ` [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating " Jakub Narebski
@ 2010-01-24 22:24 ` Petr Baudis
2010-01-25 0:03 ` Jakub Narebski
2010-01-25 11:46 ` Jakub Narebski
0 siblings, 2 replies; 60+ messages in thread
From: Petr Baudis @ 2010-01-24 22:24 UTC (permalink / raw)
To: Jakub Narebski
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
I have stupid question, common to both the original patch and this
RFC.
> [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> regenerating cache (WIP)
Just why is a "Generating..." page appropriate?
I have to admit I hate it; can you please at least make it
configurable? Why is it needed at all? It generates unnecessary
redirects, clears my previous page in the browser by a useless
placeholder, confuses non-interactive HTTP clients and I just can't
see the real purpose.
What's the gain?
Thanks,
--
Petr "Pasky" Baudis
A lot of people have my books on their bookshelves.
That's the problem, they need to read them. -- Don Knuth
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/9] gitweb: Add option to force version match
2010-01-24 21:59 ` Petr Baudis
@ 2010-01-24 23:17 ` Jakub Narebski
0 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-24 23:17 UTC (permalink / raw)
To: Petr Baudis
Cc: John 'Warthog9' Hawley, John 'Warthog9' Hawley, git
Petr Baudis <pasky@suse.cz> writes:
> On Wed, Jan 13, 2010 at 05:22:59PM -0800, John 'Warthog9' Hawley wrote:
> > +# Throw an error if git versions does not match, if $git_versions_must_match is true.
> > +if ($git_versions_must_match &&
> > + $git_version ne $version) {
> > + my $admin_contact =
> > + defined $ENV{'SERVER_ADMIN'} ? ", $ENV{'SERVER_ADMIN'}," : '';
> > + my $err_msg = <<EOT;
> > +<h1 align="center">*** Warning ***</h1>
> > +<p>
> > +This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,
> > +however git version <b>@{[esc_html($git_version)]}</b> was found on server,
> > +and administrator requested strict version checking.
+and administrator requested strict version checking by setting
+\$git_versions_must_match to false value (e.g. to 0) in gitweb configuration file,
+'@{[esc_path(-e $GITWEB_CONFIG ? $GITWEB_CONFIG : $GITWEB_CONFIG_SYSTEM)]}'.
> > +</p>
> > +<p>
> > +Please contact the server administrator${admin_contact} to either configure
> > +gitweb to allow mismatched versions, or update git or gitweb installation.
> > +</p>
> > +EOT
> > + die_error(500, 'Internal server error', $err_msg);
> > +}
> > +
>
> I get very unhappy when *I* am the server administrator and read
> messages like this. ;-) Could you please mention the
> \$git_versions_must_match in the text?
Something like the text (extension) proposed above?
Actually, J.H., if you are changing the default, you should also change the
message to mention that checking versions is the default, and administrator
can turn it off[1]. It is incorrect if $git_versions_must_match defaults
to true.
[1] And you have to take into account that gitweb config file can not
exist, and neither $GITWEB_CONFIG nor $GITWEB_CONFIG_SYSTEM needs to be
non-empty. J.H., are you really, really sure that it is a good idea to
have version check turned on by default, in light of this issue?
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-24 22:24 ` Petr Baudis
@ 2010-01-25 0:03 ` Jakub Narebski
2010-01-25 1:17 ` Jakub Narebski
2010-01-25 11:46 ` Jakub Narebski
1 sibling, 1 reply; 60+ messages in thread
From: Jakub Narebski @ 2010-01-25 0:03 UTC (permalink / raw)
To: Petr Baudis
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
On Sun, 24 Jan 2010, Petr Baudis wrote:
> I have stupid question, common to both the original patch and this
> RFC.
>
> > [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> > regenerating cache (WIP)
>
> Just why is a "Generating..." page appropriate?
>
> I have to admit I hate it; can you please at least make it
> configurable? Why is it needed at all? It generates unnecessary
> redirects, clears my previous page in the browser by a useless
> placeholder, confuses non-interactive HTTP clients and I just can't
> see the real purpose.
>
> What's the gain?
Don't ask me, I am just following original patch by John 'Warthog9'
Hawley (J.H.) here, splitting it into smaller and hopefully easier to
review parts, with each separate feature in a separate patch (so that
features like this one are more visible). I _think_ that in original
patch by J.H. this feature was configurable (at least to some extent).
As this series is more proof of concept than ready to apply patches,
some of configurability is missing currently.
I think that adding the "Generating..." page follows behavior of other
well known web applications, such as for example GMail. Currently
if caching is enabled, and there is no appropriate entry in cache (not
even [slightly] stale entry) the page is shown only when it is generated
in full: there is no streaming (although I planned to add it). So for
example for 'blame' view or for pickaxe search you could wait quite
a bit for a page to be shown. The "Generating..." page is a kind of
activity indicator here.
Note that the JavaScript-using Ajax-y version from the following patch
'[RFC PATCH 11/10] gitweb: Ajax-y "Generating..." page when
regenerating cache (WIP)'
http://permalink.gmane.org/gmane.comp.version-control.git/137896
is not visible in browser history, as it simply replaces "Generating..."
page with the result. So there is no 'unnecessary redirect', and there
is no 'useless placeholder' in browser history.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 0:03 ` Jakub Narebski
@ 2010-01-25 1:17 ` Jakub Narebski
0 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-25 1:17 UTC (permalink / raw)
To: Petr Baudis
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
Jakub Narebski wrote:
> On Sun, 24 Jan 2010, Petr Baudis wrote:
>
> > I have stupid question, common to both the original patch and this
> > RFC.
> >
> > > [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> > > regenerating cache (WIP)
> >
> > Just why is a "Generating..." page appropriate?
> >
> > I have to admit I hate it; can you please at least make it
> > configurable? Why is it needed at all? It generates unnecessary
> > redirects, clears my previous page in the browser by a useless
> > placeholder, confuses non-interactive HTTP clients and I just can't
> > see the real purpose.
> Note that the JavaScript-using Ajax-y version from the following patch
> '[RFC PATCH 11/10] gitweb: Ajax-y "Generating..." page when
> regenerating cache (WIP)'
> http://permalink.gmane.org/gmane.comp.version-control.git/137896
> is not visible in browser history, as it simply replaces "Generating..."
> page with the result. So there is no 'unnecessary redirect', and there
> is no 'useless placeholder' in browser history.
Actually I have checked and the version used on git.kernel.org does show
"Generating..." page, but it does not appear in the history. I guess
that refresh with no URL does not count as redirect.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-24 22:14 ` Petr Baudis
@ 2010-01-25 1:47 ` Jakub Narebski
2010-01-25 20:48 ` J.H.
0 siblings, 1 reply; 60+ messages in thread
From: Jakub Narebski @ 2010-01-25 1:47 UTC (permalink / raw)
To: Petr Baudis, John 'Warthog9' Hawley
Cc: John 'Warthog9' Hawley, git
On Sun, Jan 24, 2010 at 23:14 +0100, Petr Baudis wrote:
> On Fri, Jan 15, 2010 at 04:43:32PM -0800, Jakub Narebski wrote:
> > Third, wouldn't it be better to use shorter variable name, e.g. $out
> > or $oh, instead of $output_handle? We would be able to align print(f)
> > statements without making lines much longer.
>
> I totally agree, I was going to comment on this too. Please use
> something very short, I think even the three characters might be pushing
> it a bit. Having long idiom will make it real pain to both read and
> write gitweb code.
>
> It would still be nice if we could have some way to avoid this hack
> entirely. Couldn't we layer PerlIO::via over STDOUT and continue to use
> argument-less print as before?
First, the original version in v2 series, the one that can be found in
42641b1 (GITWEB - File based caching layer, 2009-12-10) commit in
git://git.kernel.org/pub/scm/git/warthog9/gitweb.git gitweb-ml-v2
http://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v2
the change was even larger. It was
print <something>;
replaced by
$output .= <something>;
and of course make functions that printed sth return formatted output
instead.
Also with original version you paid the price even if caching was turned
off (not used): the whole output had to fit in memory, and there was no
streaming, which meant that the whole output had to be generated before
it could be displayed (decreasing latency).
Second, PerlIO::via is not a good solution. One solution would be to
require Capture::Tiny if caching is enabled, and just do
$out = capture {
$actions{$action}->();
}
or even simply (if it is possible with the way 'capture' is prototyped)
$out = capture $actions{$action};
Another solution would be to use PerlIO::Util to redirect output to
scalar:
*STDOUT->push_layer(scalar => \$buffer);
# ...
*STDOUT->pop_layer();
Yet another solution would be to manipulate *STDOUT directly
$stdout = *STDOUT;
*STDOUT = $buffer_fh;
...
*STDOUT = $stdout;
for which, I think, to be done correctly we would have to duplicate
(large?) parts of Capture::Tiny.
So there is a bit of rule of preservation of difficulty at work. Either
we have large patch adding explicit filehandle to all print statements
'print <sth> -> print $out <sth>' but simple code, or have smaller patch
but complicated *STDOUT manipulation, or have small patch but rely on
non-core CPAN modules present.
> I think we (well, in practice, "you" nowadays :) should be less
> perfectionist when accepting gitweb contributions, so if the answer is
> "it's too hard for me to do right now", that's fine I guess; I know I
> can't contribute the code currently. :(
True. At least two series ('categories' which I have copy locally
in http://repo.or.cz/w/git/jnareb-git.git as gitweb/category, and
'committags' by Marcel M. Cary[1]) that are not yet in gitweb because
of that...
[1]: http://thread.gmane.org/gmane.comp.version-control.git/100418/focus=133141
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-24 22:24 ` Petr Baudis
2010-01-25 0:03 ` Jakub Narebski
@ 2010-01-25 11:46 ` Jakub Narebski
2010-01-25 13:02 ` Petr Baudis
1 sibling, 1 reply; 60+ messages in thread
From: Jakub Narebski @ 2010-01-25 11:46 UTC (permalink / raw)
To: Petr Baudis
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
On Sun, 24 Jan 2010, Petr Baudis wrote:
> I have stupid question, common to both the original patch and this
> RFC.
>
> > [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> > regenerating cache (WIP)
>
> Just why is a "Generating..." page appropriate?
>
> I have to admit I hate it; can you please at least make it
> configurable? Why is it needed at all? It [...] confuses
> non-interactive HTTP clients [...]
First, if I understand the code correctly HTTP clients which do not
honor metaredirect (http-equiv refresh) would get page which looks
the following
<html>
Generating...
</html>
<html>
Gitweb page
</html>
Second, gitweb can always check User-Agent header, and serve
"Generating..." page only to web browsers:
unless (defined $cgi->user_agent() &&
$cgi->user_agent() =~ /\b(Mozilla|Opera)\b/i) {
return;
}
or something like that.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 11:46 ` Jakub Narebski
@ 2010-01-25 13:02 ` Petr Baudis
2010-01-25 13:48 ` Jakub Narebski
0 siblings, 1 reply; 60+ messages in thread
From: Petr Baudis @ 2010-01-25 13:02 UTC (permalink / raw)
To: Jakub Narebski
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:
> On Sun, 24 Jan 2010, Petr Baudis wrote:
> > I have stupid question, common to both the original patch and this
> > RFC.
> >
> > > [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> > > regenerating cache (WIP)
> >
> > Just why is a "Generating..." page appropriate?
> >
> > I have to admit I hate it; can you please at least make it
> > configurable? Why is it needed at all? It [...] confuses
> > non-interactive HTTP clients [...]
>
> First, if I understand the code correctly HTTP clients which do not
> honor metaredirect (http-equiv refresh) would get page which looks
> the following
>
> <html>
> Generating...
> </html>
> <html>
> Gitweb page
> </html>
To be clear, I conjectured it confuses non-interactive HTTP clients from
one of the TODOs in your patch series - I'm not sure about it myself. :)
> Second, gitweb can always check User-Agent header, and serve
> "Generating..." page only to web browsers:
>
> unless (defined $cgi->user_agent() &&
> $cgi->user_agent() =~ /\b(Mozilla|Opera)\b/i) {
> return;
> }
>
> or something like that.
I'm not too happy with this. What about Safari? Opera? ELinks? There's a
lot of web browsers.
Most of the issues can be worked around, but I'm not sure why to go
through all the trouble. I just personally don't see the value in having
the placeholder in there at all, to me it is distracting UI even if all
the technicalities are put aside.
But if it will be possible to turn this off eventually, it's all your
call whether to bother implementing it. :-)
--
Petr "Pasky" Baudis
A lot of people have my books on their bookshelves.
That's the problem, they need to read them. -- Don Knuth
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 13:02 ` Petr Baudis
@ 2010-01-25 13:48 ` Jakub Narebski
2010-01-25 13:56 ` Petr Baudis
2010-01-25 20:41 ` J.H.
0 siblings, 2 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-25 13:48 UTC (permalink / raw)
To: Petr Baudis
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
On Mon, Jan 25, 2010, Petr Baudis wrote:
> On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:
>> On Sun, 24 Jan 2010, Petr Baudis wrote:
>>> I have stupid question, common to both the original patch and this
>>> RFC.
>>>
>>>> [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
>>>> regenerating cache (WIP)
>>>
>>> Just why is a "Generating..." page appropriate?
>>>
>>> I have to admit I hate it; can you please at least make it
>>> configurable? Why is it needed at all? It [...] confuses
>>> non-interactive HTTP clients [...]
>> Second, gitweb can always check User-Agent header, and serve
>> "Generating..." page only to web browsers:
>>
>> unless (defined $cgi->user_agent() &&
>> $cgi->user_agent() =~ /\b(Mozilla|Opera)\b/i) {
>> return;
>> }
>>
>> or something like that.
>
> I'm not too happy with this. What about Safari? Opera? ELinks? There's a
> lot of web browsers.
The "Mozilla" part would catch all "Mozilla compatibile" web browsers,
including Firefox (and other Gecko-based web browsers), Internet Explorer,
WebKit based browsers including Safari and Chrome and Konqueror.
The "Opera" part would catch Opera.
http://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/
As to other web browsers like Elinks, Lynx, w3m, Dillo, etc.: the issue
is whether they honor '<meta http-equiv="refresh" content="0" />'.
I think it is better to stay on the safe side; it is not disaster if web
browser is not shown "Generating..." page where it could (but see
explanation below).
> Most of the issues can be worked around, but I'm not sure why to go
> through all the trouble. I just personally don't see the value in having
> the placeholder in there at all, to me it is distracting UI even if all
> the technicalities are put aside.
The issue that "Generating..." page tries to solve is, I think, the
following.
Some actions, like 'blame' view or pickaxe search, or grep search,
can take quite a long time to generate, with times counted in
tens of seconds. It is not that visible for non-caching, because
gitweb streams output so we have at least _some_ output upfront quite
fast.
Now those patches (mine and J.H. both) make gitweb use locking
(it is IIRC configurable in J.H. patch) to make only one process
generate the page if it is missing from cache, or is stale. Now
if it is missing, we have to wait until it is generated in full
before being able to show it to client. While it is possible to
"tee" output (using PerlIO::tee, or Capture::Tiny, or tie like
CGI::Cache) writing it simultaneously to browser and to cache for
the process that is generating data, it is as far as I understand
it impossible for processes which are waiting for data. Therefore
the need for "Generating..." page, so the user does not think that
web server hung or something, and is not generating output.
We can try to reduce occurrences of cache miss stampedes by using
'expires_variance' feature[1] from CHI - Unified caching interface.
We can also turn off locking and tee output to have some output upfront
as an activity indicator instead of this "Generating..." page.
[1]: http://search.cpan.org/~jswartz/CHI-0.33/lib/CHI.pm#set
> But if it will be possible to turn this off eventually, it's all your
> call whether to bother implementing it. :-)
In my implementation it is (or rather would be) as simple as just
not passing 'generating_info' => \&git_generating_data_html in the
GitwebCache::SimpleFileCache constructor.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 13:48 ` Jakub Narebski
@ 2010-01-25 13:56 ` Petr Baudis
2010-01-25 20:32 ` J.H.
2010-01-25 20:58 ` Jakub Narebski
2010-01-25 20:41 ` J.H.
1 sibling, 2 replies; 60+ messages in thread
From: Petr Baudis @ 2010-01-25 13:56 UTC (permalink / raw)
To: Jakub Narebski
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:
> Now those patches (mine and J.H. both) make gitweb use locking
> (it is IIRC configurable in J.H. patch) to make only one process
> generate the page if it is missing from cache, or is stale. Now
> if it is missing, we have to wait until it is generated in full
> before being able to show it to client. While it is possible to
> "tee" output (using PerlIO::tee, or Capture::Tiny, or tie like
> CGI::Cache) writing it simultaneously to browser and to cache for
> the process that is generating data, it is as far as I understand
> it impossible for processes which are waiting for data. Therefore
> the need for "Generating..." page, so the user does not think that
> web server hung or something, and is not generating output.
Ah, ok, so the message is there to cover up for a technical problem. ;-)
I didn't quite realize. Then, it would be great to tweak the mechanisms
so that the user does not really have to wait.
So, I wonder about two things:
(i) How often does it happen that two requests for the same page are
received? Has anyone measured it? Or is at least able to make
a minimally educated guess? IOW, isn't this premature optimization?
(ii) Can't the locked gitwebs do the equivalent of tail -f?
P.S.: Again the disclaimer - if this is "too hard", it's better to
accept patches like they are, then improve this later. But perhaps
a better solution would be not to clutter the code by optimizing this
case at all if it's not clear it really matters in the real world.
Petr "Pasky" Baudis
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 13:56 ` Petr Baudis
@ 2010-01-25 20:32 ` J.H.
2010-01-26 1:49 ` Jakub Narebski
2010-01-28 17:39 ` Petr Baudis
2010-01-25 20:58 ` Jakub Narebski
1 sibling, 2 replies; 60+ messages in thread
From: J.H. @ 2010-01-25 20:32 UTC (permalink / raw)
To: Petr Baudis; +Cc: Jakub Narebski, git, John 'Warthog9' Hawley
On 01/25/2010 05:56 AM, Petr Baudis wrote:
> On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:
>> Now those patches (mine and J.H. both) make gitweb use locking
>> (it is IIRC configurable in J.H. patch) to make only one process
>> generate the page if it is missing from cache, or is stale. Now
>> if it is missing, we have to wait until it is generated in full
>> before being able to show it to client. While it is possible to
>> "tee" output (using PerlIO::tee, or Capture::Tiny, or tie like
>> CGI::Cache) writing it simultaneously to browser and to cache for
>> the process that is generating data, it is as far as I understand
>> it impossible for processes which are waiting for data. Therefore
>> the need for "Generating..." page, so the user does not think that
>> web server hung or something, and is not generating output.
>
> Ah, ok, so the message is there to cover up for a technical problem. ;-)
> I didn't quite realize. Then, it would be great to tweak the mechanisms
> so that the user does not really have to wait.
No, that is an incorrect assumption on how the 'Generating...' page
works, and your missing a bit of the point.
(1) The message itself 'Generating...' is a que to the user that
something is happening and that the browser is not actually hanging.
Web users are at the point where if things are not instantaneous and
show immediately they will either browse away completely or hit the
refresh button incessantly until content does appear. While the page is
usually only seen for about a second, and I'll admit it can be annoying,
it's nothing more than a 'sit tight a second'. For things like the
front page it can take upwards of 7 seconds to generate for a single
user, a lot to ask for a no response scenario.
(2) It prevents the stampeding herd problem, which was very vehemently
discussed 4 years ago by HPA and myself and roughly boils down to this:
When a single user comes into the site, in particular the front page, it
kicks off a process that will start to generate at it, causing a huge
amount of git requests into individual repositories and a lot of disk
i/o. A second user will then come in and the same requests will start
to be done from the beginning again, and so on until you basically kill
the machine because the disk i/o goes up enough that it can't ever
service the requests fast enough.
This does 2 things in the end:
1) means there's only 1 copy of the page ever being generated, thus
meaning there isn't extraneous and dangerous disk i/o going on on the system
2) prevents a user from reporting to the website that it's broken by
giving them a visual que that things aren't broken.
> So, I wonder about two things:
>
> (i) How often does it happen that two requests for the same page are
> received? Has anyone measured it? Or is at least able to make
> a minimally educated guess? IOW, isn't this premature optimization?
For most pages, not many but it happens more often than you think. The
data I have is much too old to be useful now but the front page could,
at times, have up to 30 people waiting for it without caching. This is
a very important patch believe it or not. For a site the size of
kernel.org it cannot exist without this.
But here's a quick stat, in 36 hours git.kernel.org has had
156099 accesses world wide or about 1.2 accesses a second.
android.git.kernel.org, in the same time period has had 115818 accesses.
If the first request takes 7 seconds to generate, by the time it's done
there are now 3 additional requests running. If it again takes 7
seconds to generate there are now another 3 requests running, etc. Very
quickly you've got so much i/o running the box more or less is useless.
> (ii) Can't the locked gitwebs do the equivalent of tail -f?
Not really going to help much, most of the gitweb operations won't
output much of anything beyond the header until it's collected all of
the data it needs anyway and then there will be a flurry of output. It
also means that this 'Generating...' page will only work for caching
schemes that tail can read out of, which I'm not sure it would work all
that well with things like memcached or a non-custom caching layer where
we don't necessarily have direct access to the file being written to.
At least the way I had it (and I'll admit I haven't read through Jakub's
re-working of my patches so I don't know if it's still there) is that
with background caching you only get the 'Generating...' page if it's
new or the content is grossly out of data. If it's a popular page and
it's not grossly out of date it shows you the 'stale' data while it
generates the new content in the background anyway, only locking you out
when the new file is being written. Or at least that's how I had it.
- John 'Warthog9' Hawley
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 13:48 ` Jakub Narebski
2010-01-25 13:56 ` Petr Baudis
@ 2010-01-25 20:41 ` J.H.
2010-01-26 2:30 ` Jakub Narebski
1 sibling, 1 reply; 60+ messages in thread
From: J.H. @ 2010-01-25 20:41 UTC (permalink / raw)
To: Jakub Narebski; +Cc: Petr Baudis, git, John 'Warthog9' Hawley
On 01/25/2010 05:48 AM, Jakub Narebski wrote:
> On Mon, Jan 25, 2010, Petr Baudis wrote:
>> On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:
>>> On Sun, 24 Jan 2010, Petr Baudis wrote:
>
>>>> I have stupid question, common to both the original patch and this
>>>> RFC.
>>>>
>>>>> [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
>>>>> regenerating cache (WIP)
>>>>
>>>> Just why is a "Generating..." page appropriate?
>>>>
>>>> I have to admit I hate it; can you please at least make it
>>>> configurable? Why is it needed at all? It [...] confuses
>>>> non-interactive HTTP clients [...]
>
>>> Second, gitweb can always check User-Agent header, and serve
>>> "Generating..." page only to web browsers:
>>>
>>> unless (defined $cgi->user_agent() &&
>>> $cgi->user_agent() =~ /\b(Mozilla|Opera)\b/i) {
>>> return;
>>> }
>>>
>>> or something like that.
>>
>> I'm not too happy with this. What about Safari? Opera? ELinks? There's a
>> lot of web browsers.
>
> The "Mozilla" part would catch all "Mozilla compatibile" web browsers,
> including Firefox (and other Gecko-based web browsers), Internet Explorer,
> WebKit based browsers including Safari and Chrome and Konqueror.
> The "Opera" part would catch Opera.
> http://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/
>
> As to other web browsers like Elinks, Lynx, w3m, Dillo, etc.: the issue
> is whether they honor '<meta http-equiv="refresh" content="0" />'.
> I think it is better to stay on the safe side; it is not disaster if web
> browser is not shown "Generating..." page where it could (but see
> explanation below).
Most of them do, that particular tag has been around for a long time and
since it doesn't require Javascript to do the page refresh it's pretty
much universal.
The problem is going to be with things like wget when someone wants to
snag a binary file. This works fine if the file is already cached, but
the user doesn't get what they are expecting if they get a blob that
isn't the final file, but the html contents of the page. I don't know
of any hint that things like wget would send to the server that you
could switch based on, but it would be more or less the non-background
caching state.
>> Most of the issues can be worked around, but I'm not sure why to go
>> through all the trouble. I just personally don't see the value in having
>> the placeholder in there at all, to me it is distracting UI even if all
>> the technicalities are put aside.
>
> The issue that "Generating..." page tries to solve is, I think, the
> following.
>
> Some actions, like 'blame' view or pickaxe search, or grep search,
> can take quite a long time to generate, with times counted in
> tens of seconds. It is not that visible for non-caching, because
> gitweb streams output so we have at least _some_ output upfront quite
> fast.
>
> Now those patches (mine and J.H. both) make gitweb use locking
> (it is IIRC configurable in J.H. patch) to make only one process
> generate the page if it is missing from cache, or is stale. Now
> if it is missing, we have to wait until it is generated in full
> before being able to show it to client. While it is possible to
> "tee" output (using PerlIO::tee, or Capture::Tiny, or tie like
> CGI::Cache) writing it simultaneously to browser and to cache for
> the process that is generating data, it is as far as I understand
> it impossible for processes which are waiting for data. Therefore
> the need for "Generating..." page, so the user does not think that
> web server hung or something, and is not generating output.
>
> We can try to reduce occurrences of cache miss stampedes by using
> 'expires_variance' feature[1] from CHI - Unified caching interface.
> We can also turn off locking and tee output to have some output upfront
> as an activity indicator instead of this "Generating..." page.
>
> [1]: http://search.cpan.org/~jswartz/CHI-0.33/lib/CHI.pm#set
>
>> But if it will be possible to turn this off eventually, it's all your
>> call whether to bother implementing it. :-)
>
> In my implementation it is (or rather would be) as simple as just
> not passing 'generating_info' => \&git_generating_data_html in the
> GitwebCache::SimpleFileCache constructor.
>
At least in mine it was don't allow background caching. It would force
everyone to wait on the one process that was actually generating
content. But it means a few blank pages with a spinning working icon
until the cache releases it's exclusive lock.
- John 'Warthog9' Hawley
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-25 1:47 ` Jakub Narebski
@ 2010-01-25 20:48 ` J.H.
2010-01-25 21:48 ` Jakub Narebski
0 siblings, 1 reply; 60+ messages in thread
From: J.H. @ 2010-01-25 20:48 UTC (permalink / raw)
To: Jakub Narebski; +Cc: Petr Baudis, John 'Warthog9' Hawley, git
> So there is a bit of rule of preservation of difficulty at work. Either
> we have large patch adding explicit filehandle to all print statements
> 'print <sth> -> print $out <sth>' but simple code, or have smaller patch
> but complicated *STDOUT manipulation, or have small patch but rely on
> non-core CPAN modules present.
I think depending in non-core CPAN modules is a really bad idea, and
will cause some concern and consternation with mainline distributions,
besides making it more complicated for users to get this up and running
quickly and easily. While I agree there are other ways of handling this
I think just adding the filehandle to the print statements provides the
easiest cross section of usability and functionality for everything
involved.
- John 'Warthog9' Hawley
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 13:56 ` Petr Baudis
2010-01-25 20:32 ` J.H.
@ 2010-01-25 20:58 ` Jakub Narebski
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-25 20:58 UTC (permalink / raw)
To: Petr Baudis
Cc: git, John 'Warthog9' Hawley, John 'Warthog9' Hawley
On Mon, Jan 25, 2010 at 14:56 +0100, Petr Baudis wrote:
> On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:
> > Now those patches (mine and J.H. both) make gitweb use locking
> > (it is IIRC configurable in J.H. patch) to make only one process
> > generate the page if it is missing from cache, or is stale. Now
> > if it is missing, we have to wait until it is generated in full
> > before being able to show it to client. While it is possible to
> > "tee" output (using PerlIO::tee, or Capture::Tiny, or tie like
> > CGI::Cache) writing it simultaneously to browser and to cache for
> > the process that is generating data, it is as far as I understand
> > it impossible for processes which are waiting for data. Therefore
> > the need for "Generating..." page, so the user does not think that
> > web server hung or something, and is not generating output.
>
> Ah, ok, so the message is there to cover up for a technical problem. ;-)
> I didn't quite realize. Then, it would be great to tweak the mechanisms
> so that the user does not really have to wait.
Well, the mechanism would certainly be configurable in final version
(current split version is more of proof of concept of splitting).
>
> So, I wonder about two things:
>
> (i) How often does it happen that two requests for the same page are
> received? Has anyone measured it? Or is at least able to make
> a minimally educated guess? IOW, isn't this premature optimization?
To be more exact the question is how often second request for the same
page appears when earlier request didn't finished processing. It is
the matter of both frequency of given requests, and time it takes to
generate request (which grows with growing load on server).
As to measurements: Pasky, do you have access logs, or their analysis
a la AWStats, Webalizer and the like, for repo.or.cz? Warthog9, do
you have access logs or analysis for git.kernel.org? Can you get similar
from fedorahosted?
> (ii) Can't the locked gitwebs do the equivalent of tail -f?
Well, it could, in principle, but it would need some changes.
First, instead of using temporary file to create cache entry atomically
(write to temporary file, then rename) the process generating data would
have to write to file other processes can read from. It could be e.g.
lockfile.
Second, there would be needed extended cache API so that generated data
is streamed to cache file, ->set($key, $data) ==> ->set($key, $fh) or
->set_io($key, $fh). This would mean some complications, but what might
be more important is that this trick would not work as far as I can see
with other caching backends / caching engines that the one from
gitweb/cache.pm (like memcached or mmap based ones).
Then the code could look like the following (in pseudocode):
try to acquire writers lock
if (acquired writers lock) {
generate and "tee" response
create cache entry
} else {
# <<<<<<
while (not acquired writers lock &&
sysread something) {
print <data>;
}
# >>>>>>
retrieve and print (rest) of data
}
where parts between <<<<<< and >>>>>> are new.
But there is another complication: gitweb needs to be able to deal with
the situation where process generating data got interrupted before
creating full output, or process generating data ran die_error which
does not generate any cache entry (e.g. if the URL we are trying to
access returns 404 not found - the check for existence of object can
take a while if the system is busy, I think).
Now in current implementation either cache entry is written in full, or it
is not written at all. It would be, I think, fairly easy to check with the
current code whether cache entry got generated when we acquired readers
lock (when the process get terminated, the lock gets released, which
is advantage over using atomic creating file with O_EXCL for locking),
and if we didn't repeat the whole process. With the "tee"/"tail"
solution if the process generating data got interrupted before end,
we can detect such situation, but currently I have no idea what should
be done in such situation. We can as easily as for the current solution
(which needs "Generating..." page for activity indicator) to detect
die_error situation, and with some care i.e. with not writing to cache
file directly we can ensure that cache entries contain full, correctly
generated data.
>
> P.S.: Again the disclaimer - if this is "too hard", it's better to
> accept patches like they are, then improve this later. But perhaps
> a better solution would be not to clutter the code by optimizing this
> case at all if it's not clear it really matters in the real world.
See above.
P.S. I have noticed that with current implementation (well, I am not sure
if it is true also for J.H. implementation) there is problem if there is
more than one process trying to request URL which result in die_error being
called. The design decision, present in original patch, was to not cache
"die_error" / non-"200 OK" pages; it seems sane, but I don't know if it
was a correct decision. The solution for interrupted generating process,
described above, works also for die_error pages, although it makes
die_error pages slower for such (hopefully rare) situation of simultaneous
errorneous request.
P.P.S. Both Pasky's approach to caching projects_list page, and Lea Wiemann
work on "gitweb caching" project for Google Summer of Code 2008 approached
caching in different way: by caching (parsed) data, not by caching output.
Note however that for some actions like 'snapshot' we would probably want
to have response/output caching anyway. Also for output caching we can
use X-Sendfile (or like) extension.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle
2010-01-25 20:48 ` J.H.
@ 2010-01-25 21:48 ` Jakub Narebski
0 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-25 21:48 UTC (permalink / raw)
To: J.H.; +Cc: Petr Baudis, John 'Warthog9' Hawley, git
On Mon, 25 Jan 2010, J.H. wrote:
> > So there is a bit of rule of preservation of difficulty at work. Either
> > we have large patch adding explicit filehandle to all print statements
> > 'print <sth> -> print $out <sth>' but simple code, or have smaller patch
> > but complicated *STDOUT manipulation, or have small patch but rely on
> > non-core CPAN modules present.
>
> I think depending in non-core CPAN modules is a really bad idea, and
> will cause some concern and consternation with mainline distributions,
> besides making it more complicated for users to get this up and running
> quickly and easily.
Note: the modules in question are Capture::Tiny or PerlIO::Util.
Capture::Tiny is available as perl-Capture-Tiny RPM module for Fedora Core
up from FC11 and in DAG repository, at least.
First, installing non-core CPAN modules locally for user is very simple
thanks to local::lib. After setup it is simply 'cpan -i PerlIO::Util',
and PerlIO::Util gets installed locally, by default inside ~/perl5.
Second, we can go the route that Git.pm did, namely include sources for
extra modules (private-Error.pm in case of Git.pm) and install it if
needed. With proper build system (Module::Install, Module::Build) it is
even simpler with inc or inc::latest. And there is always PAR[1] if you
want to have everything in one file.
[1]: http://par.perl.org
> While I agree there are other ways of handling this
> I think just adding the filehandle to the print statements provides the
> easiest cross section of usability and functionality for everything
> involved.
Well, manipulating *STDOUT shouldn't be, I think, much harder, although
I guess it would need more care (so that e.g. "Generating..." page gets
written to real STDOUT).
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 20:32 ` J.H.
@ 2010-01-26 1:49 ` Jakub Narebski
2010-01-28 17:39 ` Petr Baudis
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-26 1:49 UTC (permalink / raw)
To: J.H.; +Cc: Petr Baudis, git, John 'Warthog9' Hawley
On Mon, 25 Jan 2010, J.H. wrote:
> On 01/25/2010 05:56 AM, Petr Baudis wrote:
>> On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:
>>> Now those patches (mine and J.H. both) make gitweb use locking
>>> (it is IIRC configurable in J.H. patch) to make only one process
>>> generate the page if it is missing from cache, or is stale. Now
>>> if it is missing, we have to wait until it is generated in full
>>> before being able to show it to client. While it is possible to
>>> "tee" output (using PerlIO::tee, or Capture::Tiny, or tie like
>>> CGI::Cache) writing it simultaneously to browser and to cache for
>>> the process that is generating data, it is as far as I understand
>>> it impossible for processes which are waiting for data. Therefore
>>> the need for "Generating..." page, so the user does not think that
>>> web server hung or something, and is not generating output.
>>
>> Ah, ok, so the message is there to cover up for a technical problem. ;-)
>> I didn't quite realize. Then, it would be great to tweak the mechanisms
>> so that the user does not really have to wait.
>
> No, that is an incorrect assumption on how the 'Generating...' page
> works, and your missing a bit of the point.
>
> (1) The message itself 'Generating...' is a cue to the user that
> something is happening and that the browser is not actually hanging.
> Web users are at the point where if things are not instantaneous and
> show immediately they will either browse away completely or hit the
> refresh button incessantly until content does appear. While the page is
> usually only seen for about a second, and I'll admit it can be annoying,
> it's nothing more than a 'sit tight a second'. For things like the
> front page it can take upwards of 7 seconds to generate for a single
> user, a lot to ask for a no response scenario.
Pasky is asking later whether instead of separate "Generating..." page
(be it server side using trick with http-equiv refresh taking place when
page/connection is closed, or JavaScript Ajax-y one) couldn't gitweb
use streaming and have early parts of generated page (in particular the
page header and navigation bar) serve as activity indicator. I think
it is possible, if a bit more complicated, and such feature can be
introduced in a separate commit.
As to 'sit tight for a second': the code can be modified to print/show
"Generating..." page only if we are waiting more than second (or more
than half of second with utime from Time::HiRes) for the data. Pasky,
would such change address some of your concerns about "Generating..."
page as activity indicator?
>
> (2) It prevents the stampeding herd problem, which was very vehemently
> discussed 4 years ago by HPA and myself and roughly boils down to this:
>
> When a single user comes into the site, in particular the front page, it
> kicks off a process that will start to generate at it, causing a huge
> amount of git requests into individual repositories and a lot of disk
> i/o. A second user will then come in and the same requests will start
> to be done from the beginning again, and so on until you basically kill
> the machine because the disk i/o goes up enough that it can't ever
> service the requests fast enough.
As you can see from the way my proof of concept (of splitting) caching
series is split, I think that those two features are separate. Using
locking to have one writer, and protect against cache miss stampede
problem is one issue, and having "Generating..." page is another issue.
They are not orthogonal as they are now, as "Generating..." page depends
on locking to check if page got generated, but it could simply check
whether the cache file is present already.
>
> This does 2 things in the end:
>
> 1) means there's only 1 copy of the page ever being generated, thus
> meaning there isn't extraneous and dangerous disk i/o going on on the system
>
> 2) prevents a user from reporting to the website that it's broken by
> giving them a visual cue that things aren't broken.
1) is about locking ('expires_variance' can help there, too), 2) is about
"Generating..." but could be done with streaming: "tee"/"tail" solution.
>> So, I wonder about two things:
>>
>> (i) How often does it happen that two requests for the same page are
>> received? Has anyone measured it? Or is at least able to make
>> a minimally educated guess? IOW, isn't this premature optimization?
>
> For most pages, not many but it happens more often than you think. The
> data I have is much too old to be useful now but the front page could,
> at times, have up to 30 people waiting for it without caching. This is
> a very important patch believe it or not. For a site the size of
> kernel.org it cannot exist without this.
>
> But here's a quick stat, in 36 hours git.kernel.org has had
> 156099 accesses world wide or about 1.2 accesses a second.
>
> android.git.kernel.org, in the same time period has had 115818 accesses.
>
> If the first request takes 7 seconds to generate, by the time it's done
> there are now 3 additional requests running. If it again takes 7
> seconds to generate there are now another 3 requests running, etc. Very
> quickly you've got so much i/o running the box more or less is useless.
On the other hand do cache for front page (projects list) ever expire
so that there are no stale data to serve? So the issue with growing I/O
is I guess only with priming the cache.
BTW. I wonder how well the 'expires_variance' solution used in CHI would
protect against cache miss stampedes; that probably depends on the ratio
of variance to time it takes gitweb to generate the page.
>> (ii) Can't the locked gitwebs do the equivalent of tail -f?
>
> Not really going to help much, most of the gitweb operations won't
> output much of anything beyond the header until it's collected all of
> the data it needs anyway and then there will be a flurry of output.
Well, it might be enough to have headers as activity indicator... but
on the other hand users can think that server is broken if they see no
further data, as opposed to the "Generating..." page.
> It also means that this 'Generating...' page will only work for caching
> schemes that tail can read out of, which I'm not sure it would work all
> that well with things like memcached or a non-custom caching layer where
> we don't necessarily have direct access to the file being written to.
I don't think that caching interfaces such as CHI or Cache::Cache, or
individual caching engines such as Cache::Memcached or Cache::FastMmap
offer streaming access. OTOH we can read from the side as the data
is generated ("tee" to a temporary file, or something), but it is even
more complication.
>
> At least the way I had it (and I'll admit I haven't read through Jakub's
> re-working of my patches so I don't know if it's still there) is that
> with background caching you only get the 'Generating...' page if it's
> new or the content is grossly out of data. If it's a popular page and
> it's not grossly out of date it shows you the 'stale' data while it
> generates the new content in the background anyway, only locking you out
> when the new file is being written. Or at least that's how I had it.
It is there, i.e. gitweb would serve stale data if it can be found,
but it lacks 'too stale to serve as stale data' timeout/expire time.
I skipped this feature to get split series out of door faster.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 20:41 ` J.H.
@ 2010-01-26 2:30 ` Jakub Narebski
0 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-26 2:30 UTC (permalink / raw)
To: J.H.; +Cc: Petr Baudis, git, John 'Warthog9' Hawley
On Mon, 25 Jan 2010, J.H. wrote:
> On 01/25/2010 05:48 AM, Jakub Narebski wrote:
>> On Mon, Jan 25, 2010, Petr Baudis wrote:
>>> On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:
>>>> On Sun, 24 Jan 2010, Petr Baudis wrote:
>>
>>>>> I have stupid question, common to both the original patch and this
>>>>> RFC.
>>>>>
>>>>>> [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
>>>>>> regenerating cache (WIP)
>>>>>
>>>>> Just why is a "Generating..." page appropriate?
>>>>>
>>>>> I have to admit I hate it; can you please at least make it
>>>>> configurable? Why is it needed at all? It [...] confuses
>>>>> non-interactive HTTP clients [...]
>>
>>>> Second, gitweb can always check User-Agent header, and serve
>>>> "Generating..." page only to web browsers:
>>>>
>>>> unless (defined $cgi->user_agent() &&
>>>> $cgi->user_agent() =~ /\b(Mozilla|Opera)\b/i) {
>>>> return;
>>>> }
>>>>
>>>> or something like that.
>>>
>>> I'm not too happy with this. What about Safari? Opera? ELinks? There's a
>>> lot of web browsers.
>>
>> The "Mozilla" part would catch all "Mozilla compatibile" web browsers,
>> including Firefox (and other Gecko-based web browsers), Internet Explorer,
>> WebKit based browsers including Safari and Chrome and Konqueror.
>> The "Opera" part would catch Opera.
>> http://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/
>>
>> As to other web browsers like Elinks, Lynx, w3m, Dillo, etc.: the issue
>> is whether they honor '<meta http-equiv="refresh" content="0" />'.
>> I think it is better to stay on the safe side; it is not disaster if web
>> browser is not shown "Generating..." page where it could (but see
>> explanation below).
>
> Most of them do, that particular tag has been around for a long time and
> since it doesn't require Javascript to do the page refresh it's pretty
> much universal.
For this method to work there is additional assumption about web browser,
namely that it would follow refresh only after it gets page in full (only
when connection closes). But perhaps that is true for all web browsers
following http-equiv refresh.
>
> The problem is going to be with things like wget when someone wants to
> snag a binary file. This works fine if the file is already cached, but
> the user doesn't get what they are expecting if they get a blob that
> isn't the final file, but the html contents of the page. I don't know
> of any hint that things like wget would send to the server that you
> could switch based on, but it would be more or less the non-background
> caching state.
Well, the above solution would work as long as you don't run wget with
--user-agent.
[...]
>>> But if it will be possible to turn this off eventually, it's all your
>>> call whether to bother implementing it. :-)
>>
>> In my implementation it is (or rather would be) as simple as just
>> not passing 'generating_info' => \&git_generating_data_html in the
>> GitwebCache::SimpleFileCache constructor.
>
> At least in mine it was don't allow background caching. It would force
> everyone to wait on the one process that was actually generating
> content. But it means a few blank pages with a spinning working icon
> until the cache releases it's exclusive lock.
Theoretically at least having "Generating..." page is orthogonal to using
locking, to background serving and to serving stale data. In current
implementation those features are dependent, serving stale data on locking,
and "Generating..." also on locking.
I think that serving stale data should depend on background caching, as
otherwise we introduce inequality in treating process which regenerates
cache (and must wait without background caching) and rest of processes
(which are served stale data).
Certainly in final version of patch all this should be made configurable.
P.S. With background caching there is a question whether parent should
generate and child display stale or "Generating...", or whether parent
should display stale/activity and child (re)generate cache entry. Also
there is an issue whether parent should waitpid for child, would fiddling
with SIGCHLD handler be enough, or is it not needed.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-25 20:32 ` J.H.
2010-01-26 1:49 ` Jakub Narebski
@ 2010-01-28 17:39 ` Petr Baudis
2010-01-31 11:58 ` Jakub Narebski
1 sibling, 1 reply; 60+ messages in thread
From: Petr Baudis @ 2010-01-28 17:39 UTC (permalink / raw)
To: J.H.; +Cc: Jakub Narebski, git, John 'Warthog9' Hawley
On Mon, Jan 25, 2010 at 12:32:37PM -0800, J.H. wrote:
> This does 2 things in the end:
>
> 1) means there's only 1 copy of the page ever being generated, thus
> meaning there isn't extraneous and dangerous disk i/o going on on the system
But this has nothing to do with what you _do_ when there are multiple
requests, whether you do the same as if caching was disabled (hang until
content is generated) or doing something novel (creating redirects
through "Generating..." page).
> 2) prevents a user from reporting to the website that it's broken by
> giving them a visual que that things aren't broken.
But this has nothing to do with caching per se, right? I think it
actually makes _no difference_ if caching is enabled or not to this
problem, or am I missing something?
My point is, I guess, that showing the Generating page doesn't seem to
have actually anything to do with the caching itself?
> > So, I wonder about two things:
> >
> > (i) How often does it happen that two requests for the same page are
> > received? Has anyone measured it? Or is at least able to make
> > a minimally educated guess? IOW, isn't this premature optimization?
>
> For most pages, not many but it happens more often than you think. The
> data I have is much too old to be useful now but the front page could,
> at times, have up to 30 people waiting for it without caching. This is
> a very important patch believe it or not. For a site the size of
> kernel.org it cannot exist without this.
..snip numbers..
Ok, I think you convinced me here (even if I think the front page is
kind of special case).
> > (ii) Can't the locked gitwebs do the equivalent of tail -f?
>
> Not really going to help much, most of the gitweb operations won't
> output much of anything beyond the header until it's collected all of
> the data it needs anyway and then there will be a flurry of output. It
> also means that this 'Generating...' page will only work for caching
> schemes that tail can read out of, which I'm not sure it would work all
> that well with things like memcached or a non-custom caching layer where
> we don't necessarily have direct access to the file being written to.
>
> At least the way I had it (and I'll admit I haven't read through Jakub's
> re-working of my patches so I don't know if it's still there) is that
> with background caching you only get the 'Generating...' page if it's
> new or the content is grossly out of data. If it's a popular page and
> it's not grossly out of date it shows you the 'stale' data while it
> generates the new content in the background anyway, only locking you out
> when the new file is being written. Or at least that's how I had it.
Well, my user experience with gitweb on kernel.org is that I get
Generating... page all the time when I dive deep enough to the object
tree. I just find it really distracting and sometimes troublesome when
I want to wget some final object.
I think it's fine to take in the caching support with the Generating...
page in the bundle, but I do want to declare my intention to get rid of
it later, at least for caching backends that could do without it - for
pages where content appears incrementally, tail -f, for pages where
content appears all at once, show at least the header and some "I'm
busy" notification without redirects.
--
Petr "Pasky" Baudis
If you can't see the value in jet powered ants you should turn in
your nerd card. -- Dunbal (464142)
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating cache (WIP)
2010-01-28 17:39 ` Petr Baudis
@ 2010-01-31 11:58 ` Jakub Narebski
0 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-01-31 11:58 UTC (permalink / raw)
To: Petr Baudis; +Cc: J.H., git, John 'Warthog9' Hawley
On Thu, Jan 28, 2010, Petr Baudis wrote:
> On Mon, Jan 25, 2010 at 12:32:37PM -0800, J.H. wrote:
> > This does 2 things in the end:
> >
> > 1) means there's only 1 copy of the page ever being generated, thus
> > meaning there isn't extraneous and dangerous disk i/o going on on the system
>
> But this has nothing to do with what you _do_ when there are multiple
> requests, whether you do the same as if caching was disabled (hang until
> content is generated) or doing something novel (creating redirects
> through "Generating..." page).
>
> > 2) prevents a user from reporting to the website that it's broken by
> > giving them a visual que that things aren't broken.
>
> But this has nothing to do with caching per se, right? I think it
> actually makes _no difference_ if caching is enabled or not to this
> problem, or am I missing something?
>
>
> My point is, I guess, that showing the Generating page doesn't seem to
> have actually anything to do with the caching itself?
The point is that without caching it is easy to streaming response, and
to consider early parts of page (like page header, generated before any
heavy work) to serve as activity indicator.
With caching it is difficult to have streaming response, both from
technical point of view (writer must generate to client and to cache
simultaneously, readers must know when writer finished work to close
connection), and from robustness point of view (what happens if writer
is interrupted / killed before finishing generating output). With
"generate then display" (which is not exclusive to caching, and is
another possible way of generating content even without caching) we
rather need some kind of activity indicator like "Generating..." page.
I think that "Generating..." page can be improved in two ways:
* Show "Generating..." page only if we are waiting for response for
more than one second. This might need mucking with alarms, as I think
that sleep 1 before $self->generating_info(...) would be not a good
solution.
* Stream response (using PerlIO::tee layer from PerlIO::Util, or
Capture::Tiny module, or tied filehandle like in CGI::Cache) for
writer (i.e. process generating data), and wait for it to be finished
(perhaps with "Generating...") in readers. This way you wouldn't get
"Generating..." page for rare views/URLs, and for common views/URLs
there is high probability that you would not need "Generating..."
page as there would be slightly stale response to serve.
Of course one can implement _both_ of those solutions, i.e. wait one
seconds in readers, and stream in writer.
I am not sure, but there might be another issue why activity indicator
is more important for the case with caching enabled. If you interrupt
writer, one of readers waiting for finished data would have to take
role of writer, which besides need for technical solution to this problem
would mean longer wait.
[..]
> > > (ii) Can't the locked gitwebs do the equivalent of tail -f?
> >
> > Not really going to help much, most of the gitweb operations won't
> > output much of anything beyond the header until it's collected all of
> > the data it needs anyway and then there will be a flurry of output. It
> > also means that this 'Generating...' page will only work for caching
> > schemes that tail can read out of, which I'm not sure it would work all
> > that well with things like memcached or a non-custom caching layer where
> > we don't necessarily have direct access to the file being written to.
> >
> > At least the way I had it (and I'll admit I haven't read through Jakub's
> > re-working of my patches so I don't know if it's still there) is that
> > with background caching you only get the 'Generating...' page if it's
> > new or the content is grossly out of data. If it's a popular page and
> > it's not grossly out of date it shows you the 'stale' data while it
> > generates the new content in the background anyway, only locking you out
> > when the new file is being written. Or at least that's how I had it.
>
> Well, my user experience with gitweb on kernel.org is that I get
> "Generating..." page all the time when I dive deep enough to the object
> tree. I just find it really distracting and sometimes troublesome when
> I want to wget some final object.
First, the user_agent checking would help there (it's a pity that all
web spiders (bots) and all non-interactive downloaders do not say what
they are explicitly in User-Agent string).
Second, I guess that waiting 1 second (or more) before showing
"Generating..." page would help in most cases.
>
> I think it's fine to take in the caching support with the Generating...
> page in the bundle, but I do want to declare my intention to get rid of
> it later, at least for caching backends that could do without it - for
> pages where content appears incrementally, tail -f, for pages where
> content appears all at once, show at least the header and some "I'm
> busy" notification without redirects.
In the final version this should be fully configurable. Note that
the series of patches I have send were just proof of concept for
splitting caching patch into smaller parts / individual features.
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 00/10] gitweb: Simple file based output caching
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
` (10 preceding siblings ...)
2010-01-24 13:54 ` [RFC PATCH 11/10] gitweb: Ajax-y "Generating..." page when regenerating cache (WIP) Jakub Narebski
@ 2010-02-06 0:51 ` J.H.
2010-02-06 23:56 ` Jakub Narebski
11 siblings, 1 reply; 60+ messages in thread
From: J.H. @ 2010-02-06 0:51 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git, John 'Warthog9' Hawley
> Table of contents:
> ~~~~~~~~~~~~~~~~~~
> [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing
> for caching)
This looks fine, I did some quick testing to verify that this would work
- and it does.
The only caveat that needs to be aware is that if the layer is going to
output binary data it needs to flip the whole stream to :raw before
outputting (this is going to be more specific to the caching layer).
One advantage to having the file handles separate is that it's easier to
distinguish if the data is going to need to be binary data that will
need to be flipped properly.
Also means you could cache the binary data differently than textual data.
I.E. binary data gets saved to disk, but page data gets saved to memcached.
Just food for thought, I'm not sure which way makes more sense
personally, though I would have a tendency to err on the side of
flexibility and have both.
> [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)
note: delaying additional comment till I've finished reading through the
basics of the following patches.
> [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching
Ok this is quite the departure from what I had, I'm unsure that it's the
right way to go, but it obviously has merits (I.E. much simpler addition
of any Cache::Cache or CHI compatible caching layer)
This patch itself looks fine, and as it states it borrows heavily from
my basic implementation - just wraps it differently. I might have some
thoughts on extending this a bit to be a bit more flushed out from a
basic standpoint.
Need to dig through it some more, but I'm generally ok with it.
> [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration
Looks fine to me, though the note about getting the errors should get
moved to previous patch, as it says.
Note: I'm going to stop here as the following are WIP and I want to play
around with this particular direction on my own a little more before
further comment. There's some ideas running around I want to try and
get down in code first. Me moving on and trying these other ideas is
not a reflection on the following patches, just some alternative
thinking before I discuss some other ideas on the following patches.
Also I've been sitting on this e-mail in this state for almost a week
while I've been playing with this and having to fight other fires and I
know that Jakub has been looking for commentary on this.
> [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)
> output caching (WIP)
> [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)
> [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)
> [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'
> problem (WIP)
> [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for
> filling cache (WIP)
> [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> regenerating cache (WIP)
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 00/10] gitweb: Simple file based output caching
2010-02-06 0:51 ` [RFC PATCH 00/10] gitweb: Simple file based output caching J.H.
@ 2010-02-06 23:56 ` Jakub Narebski
2010-02-07 12:35 ` Jakub Narebski
0 siblings, 1 reply; 60+ messages in thread
From: Jakub Narebski @ 2010-02-06 23:56 UTC (permalink / raw)
To: J.H.; +Cc: git, John 'Warthog9' Hawley, Petr Baudis
On Sat, 6 Feb 2010, J.H. wrote:
> > Table of contents:
> > ~~~~~~~~~~~~~~~~~~
> > [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing
> > for caching)
I am working on v2 of this series, where this patch is not necessary.
It uses *STDOUT->push_layer(scalar => \$data) and *STDOUT->pop_layer()
from PerlIO::Util if it is available, and manipulation of *STDOUT (which
means *STDOUT = $data_fh and not $out = $data_fh). But I must say that
doing capture of STDOUT (only; STDERR is not captured) without requiring
extra Perl modules (like recommended Capture::Tiny or e.g. IO::Capture),
and especially testing that it works correctly with capturing output
of cache_fetch is serious PITA.
This patch has the advantage that all operations are simpler. In
particular it is easy to have section which should be not captured,
or where capture should be turned off (slightly different).
It has the disadvantage that all future contributions must use
"print $out <something>" / "print {$out} <something>", and that
contributions from before this change would have to be carefully
updated. (Well, we could probably add the test that would check
that everything that needs to go to $out does, and everything that
shouldn't got to $out but to STDOUT doesn't.)
If I were to have such patch in new version of "gitweb output
caching" series, I would make the following changes:
* (optionally) use simpler 'print $out <sth>' instead of visually
distinct 'print {$out} <sth>', where from first glance one can
see that $out is filehandle and not something to be printed
* use short filehandle name: $out, or $oh, or $o/$O.
* split above patch in 2 to 4 patches:
- pure mechanical (scripted) change:
+ print <sth> -> print $out <sth>
+ printf(<sth>) -> printf($out <sth>)
+ binmode STDOUT -> binmode $out
The last with possible exception of very first binmode call.
- realign (purely whitespace change)
- wrap too long lines (newlines and whitespace), optional
- change $out to $bout/$bin ($binary_output_fh) where needed;
but see comment below (optional)
>
> This looks fine, I did some quick testing to verify that this would work
> - and it does.
I have only ran test, and didn't actually check that it works correctly.
This commit shouldn't change gitweb behaviour at all.
>
> The only caveat that needs to be aware is that if the layer is going to
> output binary data it needs to flip the whole stream to :raw before
> outputting (this is going to be more specific to the caching layer).
>
> One advantage to having the file handles separate is that it's easier to
> distinguish if the data is going to need to be binary data that will
> need to be flipped properly.
I don't think that it would be needed.
First, all mode changing operations, i.e. calls to binmode are changed
to act on $out rather than on STDOUT it means. It means that if we are
using 'in memory file' to capture output to scalar variable, then captured
data would be properly converted in variable. So it would be enough to
save this variable in :raw mode to file. If we are saving directly to
cache file, then of course saved data would go through layer and would
be converted properly. In any case in cache file we would have _already_
_converted_ data.
This means that regardless whether $out used ':utf8' (pseudo)layer,
or ':raw' (pseudo)layer, if we read from cache file in ':raw' (binary mode)
and print data from cache to original (true) STDOUT also in ':raw' mode,
we would print correctly formatted data.
>
> Also means you could cache the binary data differently than textual data.
>
> I.E. binary data gets saved to disk, but page data gets saved to memcached.
That's true, but on the other hand it would be easy to add some extra
command marking data as binary below binmode. Or we can examine IO
layers (using PerlIO::get_layers($out); the PerlIO module is in Perl
core) if there is 'utf8' layer or 'raw' (pseudo)layer.
>
> Just food for thought, I'm not sure which way makes more sense
> personally, though I would have a tendency to err on the side of
> flexibility and have both.
It might be good idea... but nevertheless I'd like to have short name
for binary filehandle, if we decode to keep it. What should it be?
$bout, $bin, $B, $bin_out, $out_bin, $bin_fh?
>
> > [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)
>
> note: delaying additional comment till I've finished reading through the
> basics of the following patches.
This is to use later _full_ _normalized_ URI as cache key for given page.
IIRC in your original patch you ignored path_info; but on the other hand
git.kernel.org has path_info feature turned off...
>
> > [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching
>
> Ok this is quite the departure from what I had, I'm unsure that it's the
> right way to go, but it obviously has merits (I.E. much simpler addition
> of any Cache::Cache or CHI compatible caching layer)
>
> This patch itself looks fine, and as it states it borrows heavily from
> my basic implementation - just wraps it differently. I might have some
> thoughts on extending this a bit to be a bit more flushed out from a
> basic standpoint.
>
> Need to dig through it some more, but I'm generally ok with it.
Note that the new implementation in (not send yet) new version of
"gitweb output caching" series is based more on newer and more modern
CHI unified interface rather than older Cache::Cache interface. It
is I think much cleaner and easier to read.
The major difference from your implementation is that in my version
the gitweb caching engine uses "save to temporary file + rename file
to final name" method to have atomic write to cache (atomic cache
filling). It should be more robust, but OTOH it introduces a bit of
performance penalty. With locking and single writer we could use
predictable temporary file name rather than using tempfile/mkstemp
or equivalent from File::Temp, or UUID based filename like CHI does
it.
Also, tests.
Current code (even the v2 version) lacks proper error detection, error
signalling and logging.
>
> > [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration
>
> Looks fine to me, though the note about getting the errors should get
> moved to previous patch, as it says.
I wanted to get this series out faster, that is why it is not polished.
>
> Note: I'm going to stop here as the following are WIP and I want to play
> around with this particular direction on my own a little more before
> further comment. There's some ideas running around I want to try and
> get down in code first. Me moving on and trying these other ideas is
> not a reflection on the following patches, just some alternative
> thinking before I discuss some other ideas on the following patches.
Take a look at gitweb/cache-kernel-v2 branch (the new caching series).
Note however that it would be subject to rebasing / changes.
>
> Also I've been sitting on this e-mail in this state for almost a week
> while I've been playing with this and having to fight other fires and I
> know that Jakub has been looking for commentary on this.
Thank you very much for your commentary, in spite of your heavy load.
>
> > [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)
> > output caching (WIP)
> > [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)
> > [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)
> > [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'
> > problem (WIP)
> > [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for
> > filling cache (WIP)
> > [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when
> > regenerating cache (WIP)
There is new version of this series in gitweb/cache-kernel-v2 in my
git/jnareb-git.git fork (clone) of git.git repository at repo.or.cz.
Now all commits have proper description (for first series one had to
read comment section in emails for commit description), and all features
are tested (at least on API level, and to some extent) -- full tests
do require having PerlIO::Util installed (I have done it following
local::lib and installing it from 'cpan' client), though.
Also all features are fully configurable, to even greater extent than
in original series by J.H. (this what what v1 was lacking). And there
is (see diffstat) section about caching in gitweb/README.
The following changes since commit d5f8a3d6f4d946c33459e00edf02819f89711777:
Junio C Hamano (1):
Merge branch 'master' into next
are available in the git repository at:
git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v2
You can view it via gitweb at:
http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel-v2
http://repo.or.cz/w/git/jnareb-git.git/log/refs/heads/gitweb/cache-kernel-v2
SHORTLOG (10):
gitweb: href(..., -path_info => 0|1)
gitweb/cache.pm - Very simple file based caching
gitweb/cache.pm - Stat-based cache expiration
gitweb: Use Cache::Cache compatibile (get, set) output caching
gitweb/cache.pm - Adaptive cache expiration time
gitweb: Use CHI compatibile (compute method) caching
gitweb/cache.pm - Use locking to avoid 'cache miss stampede' problem
gitweb/cache.pm - Serve stale data when waiting for filling cache
gitweb/cache.pm - Regenerate (refresh) cache in background
gitweb: Show appropriate "Generating..." page when regenerating cache
gitweb/README | 70 +++++
gitweb/cache.pm | 527 ++++++++++++++++++++++++++++++++
gitweb/gitweb.perl | 301 +++++++++++++++++-
t/gitweb-lib.sh | 2 +
t/t9500-gitweb-standalone-no-errors.sh | 19 ++
t/t9503-gitweb-caching.sh | 32 ++
t/t9503/test_cache_interface.pl | 380 +++++++++++++++++++++++
t/test-lib.sh | 3 +
8 files changed, 1319 insertions(+), 15 deletions(-)
create mode 100644 gitweb/cache.pm
create mode 100755 t/t9503-gitweb-caching.sh
create mode 100755 t/t9503/test_cache_interface.pl
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 00/10] gitweb: Simple file based output caching
2010-02-06 23:56 ` Jakub Narebski
@ 2010-02-07 12:35 ` Jakub Narebski
0 siblings, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-02-07 12:35 UTC (permalink / raw)
To: J.H.; +Cc: git, John 'Warthog9' Hawley
On Sun, 7 Feb 2010, Jakub Narebski wrote:
> There is new version of this series in gitweb/cache-kernel-v2 in my
> git/jnareb-git.git fork (clone) of git.git repository at repo.or.cz.
> Now all commits have proper description (for first series one had to
> read comment section in emails for commit description), [...]
Below there are commit messages for gitweb/cache-kernel-v2 branch after
rebase and fixups:
commit 560e2ab10d0f8457fbeca7a26814ff3e32396f7b
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 11:27:22 2010 +0100
gitweb: href(..., -path_info => 0|1)
If named boolean option -path_info is passed to href() subroutine, use
its value to decide whether to generate path_info URL form. If this
option is not passed, href() queries 'pathinfo' feature to check
whether to generate path_info URL (if generating path_info link is
possible at all).
href(-replay=>1, -path_info=>0) is meant to be used to generate a key
for caching gitweb output; alternate solution would be to use freeze()
from Storable (core module) on %input_params hash (or its reference),
e.g.:
$key = freeze \%input_params;
or other serialization technique.
While at it document extra options/flags to href().
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/gitweb.perl | 7 ++++++-
1 files changed, 6 insertions(+), 1 deletions(-)
commit dd6e8dc27d5b799bd2a1aed03738195dfe3bc5e7
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:13:06 2010 +0100
gitweb/cache.pm - Very simple file based caching
This is first step towards implementing file based output (response)
caching layer that is used on such large sites as kernel.org.
This patch introduces GitwebCaching::SimpleFileCache package, which
follows Cache::Cache / CHI interface, although do not implement it
fully. The intent of following established convention is to be able
in the future to replace our simple file based cache e.g. by one using
memcached.
Like in original patch by John 'Warthog9' Hawley (J.H.) (the one this
commit intends to be incremental step to), the data is stored in the
case as-is, without adding metadata (like expiration date), and
without serialization (which means only scalar data).
To be implemented (from original patch by J.H.):
* cache expiration (based on file stats, current time and global
expiration time); currently elements in cache do not expire
* actually using this cache in gitweb, except error pages
* adaptive cache expiration, based on average system load
* optional locking interface, where only one process can update cache
(using flock)
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Possible extensions (beyond what was in original patch):
* (optionally) show information about cache utilization
* AJAX (JavaScript-based) progress indicator
* JavaScript code to update relative dates in cached output
* make cache size-aware (try to not exceed specified maximum size)
* utilize X-Sendfile header (or equivalent) to show cached data
(optional, as it makes sense only if web server supports sendfile
feature and have it enabled)
* variable expiration feature from CHI, allowing items to expire a bit
earlier than the stated expiration time to prevent cache miss
stampedes (although locking, if available, should take care of
this).
The code of GitwebCaching::SimpleFileCache package in gitweb/cache.pm
was heavily based on file-based cache in Cache::Cache package, i.e.
on Cache::FileCache, Cache::FileBackend and Cache::BaseCache, and on
file-based cache in CHI, i.e. on CHI::Driver::File and CHI::Driver
(including implementing atomic write, something that original patch
lacks).
This patch does not yet enable output caching in gitweb (it doesn't
have all required features yet); on the other hand it includes tests,
currently testing only cache Perl API.
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 269 +++++++++++++++++++++++++++++++++++++++
t/t9503-gitweb-caching.sh | 32 +++++
t/t9503/test_cache_interface.pl | 84 ++++++++++++
t/test-lib.sh | 3 +
4 files changed, 388 insertions(+), 0 deletions(-)
create mode 100644 gitweb/cache.pm
create mode 100755 t/t9503-gitweb-caching.sh
create mode 100755 t/t9503/test_cache_interface.pl
commit 3914e7da792fec50fcc64c0e644d54cf4451703a
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:13:17 2010 +0100
gitweb/cache.pm - Stat-based cache expiration
Add stat-based cache expiration to file-based GitwebCache::SimpleFileCache.
Contrary to the way other caching interfaces such as Cache::Cache and CHI
do it, the time cache element expires in is _global_ value associated with
cache instance, and is not local property of cache entry. (Currently cache
entry does not store any metadata associated with entry... which means that
there is no need for serialization / marshalling / freezing and thawing.)
Default expire time is -1, which means never expire.
To check if cache entry is expired, GitwebCache::SimpleFileCache compares
difference between mtime (last modify time) of a cache file and current time
with (global) time to expire. It is done using CHI-compatible is_valid()
method.
Add some tests checking that expiring works correctly (on the level of API).
To be implemented (from original patch by J.H.):
* actually using this cache in gitweb, except error pages
* adaptive cache expiration, based on average system load
* optional locking interface, where only one process can update cache
(using flock)
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 34 ++++++++++++++++++++++++++++++++--
t/t9503/test_cache_interface.pl | 10 ++++++++++
2 files changed, 42 insertions(+), 2 deletions(-)
commit a55625cb0f2d6c08a28e774fd2ddb4e5347a24b3
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:13:27 2010 +0100
gitweb: Use Cache::Cache compatible (get, set) output caching
This commit actually adds output caching to gitweb, as we have now
minimal features required for it in GitwebCache::SimpleFileCache
(a 'dumb' but fast file-based cache engine). To enable cache you need
at least set $caching_enabled to true in gitweb config, and copy cache.pm
from gitweb/ alongside gitweb.cgi - this is described in more detail
in the new "Gitweb caching" section in gitweb/README
Currently cache support related subroutines in cache.pm (which are
outside GitwebCache::SimpleFileCache package) are not well separated
from gitweb script itself; cache.pm lacks encapsulation. cache.pm
assumes that there are href() subroutine and %actions variable, and
that there exist $actions{$action} (where $action is parameter passed
to cache_fetch), and it is a code reference (see also comments in
t/t9503/test_cache_interface.pl). This is remaining artifact from the
original patch by J.H. (which also had cache_fetch() subroutine).
Gitweb itself uses directly only cache_fetch, to get page from cache
or to generate page and save it to cache, and cache_stop, to be used
in die_error subroutine, as currently error pages are not cached.
The cache_fetch subroutine captures output (from STDOUT only, as
STDERR is usually logged) using either ->push_layer()/->pop_layer()
from PerlIO::Util submodule (if it is available), or by setting and
restoring *STDOUT. Note that only the former could be tested reliably
to be reliable in t9503 test!
Enabling caching causes the following additional changes to gitweb
output:
* Disables content-type negotiation (choosing between 'text/html'
mimetype and 'application/xhtml+xml') when caching, as there is no
content-type negotiation done when retrieving page from cache.
Use 'text/html' mimetype that can be used by all browsers.
* Disable timing info (how much time it took to generate original
page, and how many git commands it took), and in its place show when
page was originally generated (in GMT / UTC timezone).
Add basic tests of caching support to t9500-gitweb-standalone-no-errors
test: set $caching_enabled to true and check for errors for first time
run (generating cache) and second time run (retrieving from cache) for a
single view - summary view for a project.
If PerlIO::Util is available (see comments), test that cache_fetch
behaves correctly, namely that it saves and restores action output in
cache, and that it prints generated output or cached output.
To be implemented (from original patch by J.H.):
* adaptive cache expiration, based on average system load
* optional locking interface, where only one process can update cache
(using flock)
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/README | 70 ++++++++++++++++++++++
gitweb/cache.pm | 78 ++++++++++++++++++++++++
gitweb/gitweb.perl | 102 ++++++++++++++++++++++++++++----
t/gitweb-lib.sh | 2 +
t/t9500-gitweb-standalone-no-errors.sh | 19 ++++++
t/t9503/test_cache_interface.pl | 93 +++++++++++++++++++++++++++++
6 files changed, 352 insertions(+), 12 deletions(-)
commit 3e471ebd31e881ce1439f23075378c2ec6b95e4d
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:13:31 2010 +0100
gitweb/cache.pm - Adaptive cache expiration time
Add to GitwebCache::SimpleFileCache support for adaptive lifetime
(cache expiration) control. Cache lifetime can be increased or
decreased by any factor, e.g. load average, through the definition
of the 'check_load' callback.
Note that using ->set_expires_in, or unsetting 'check_load' via
->set_check_load(undef) turns off adaptive caching.
Make gitweb automatically adjust cache lifetime by load, using
get_loadavg() function. Define and describe default parameters for
dynamic (adaptive) cache expiration time control.
There are some very basic tests of dynamic expiration time in t9503,
namely checking if dynamic expire time is within given upper and lower
bounds.
To be implemented (from original patch by J.H.):
* optional locking interface, where only one process can update cache
(using flock)
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 55 +++++++++++++++++++++++++++++++++++---
gitweb/gitweb.perl | 27 +++++++++++++++++-
t/t9503/test_cache_interface.pl | 22 +++++++++++++++
3 files changed, 97 insertions(+), 7 deletions(-)
commit 984390f99c33d82cd4ddbfa6e00c721d9e74cddb
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:13:52 2010 +0100
gitweb: Use CHI compatible (compute method) caching
If $cache provides CHI compatible ->compute($key, $code) method, use it
instead of Cache::Cache compatible ->get($key) and ->set($key, $data).
While at it, refactor regenerating cache into cache_calculate subroutine.
GitwebCache::SimpleFileCache provides 'compute' method, which currently
simply use 'get' and 'set' methods in proscribed manner. Nevertheless
'compute' method can be more flexible in choosing when to refresh cache,
and which process is to refresh/(re)generate cache entry. This method
would use (advisory) locking to prevent 'cache miss stampede' (aka
'stampeding herd') problem in the next commit.
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 39 ++++++++++++++++++++++++++++++++++++---
1 files changed, 36 insertions(+), 3 deletions(-)
commit 7d0109e4379f5187364edf7c25cdbc5247609f64
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:18:14 2010 +0100
gitweb/cache.pm - Use locking to avoid 'cache miss stampede' problem
In the ->compute($key, $code) method from GitwebCache::SimpleFileCache,
use locking (via flock) to ensure that only one process would generate
data to update/fill-in cache; the rest would wait for the cache to
be (re)generated and would read data from cache.
Currently this feature can not be disabled (via %cache_options).
A test in t9503 shows that in the case where there are two clients
trying to simultaneously access non-existent or stale cache entry,
(and generating data takes (artifically) a bit of time), if they are
using ->compute method the data is (re)generated once, as opposed to
if those clients are just using ->get/->set methods.
To be implemented (from original patch by J.H.):
* background building, and showing stale cache
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 29 ++++++++++++++++-
t/t9503/test_cache_interface.pl | 65 +++++++++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+), 2 deletions(-)
commit e7985f69eb9000860b155939d5fd7040e30f682f
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:19:21 2010 +0100
gitweb/cache.pm - Serve stale data when waiting for filling cache
When process fails to acquire exclusive (writers) lock, then instead
of waiting for the other process to (re)generate and fill cache, serve
stale (expired) data from cache. This is of course possible only if
there is some stale data in cache for given key.
This feature of GitwebCache::SimpleFileCache is used only for an
->update($key, $code) method. It is controlled by 'max_lifetime'
cache parameter; you can set it to -1 to always serve stale data
if it exists, and you can set it to 0 (or any value smaller than
'expires_min') to turn this feature off.
This feature, as it is implemented currently, makes ->update() method a
bit assymetric with respect to process that acquired writers lock and
those processes that didn't, which can be seen in the new test in t9503.
The process that is to regenerate (refresh) data in cache must wait for
the data to be generated in full before showing anything to client, while
the other processes show stale (expired) data immediately. In order to
remove or reduce this assymetry gitweb would need to employ one of the two
alternate solutions. Either data should be (re)generated in background,
so that process that acquired writers lock would generate data in
background while serving stale data, or alternatively the process that
generates data should pass output to original STDOUT while capturing it
("tee" otput).
When developing this feature, ->is_valid() method acquired additional
extra optional parameter, where one cap pass expire time instead of using
cache-wode global expire time.
To be implemented (from original patch by J.H.):
* background building,
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 23 ++++++++++----
gitweb/gitweb.perl | 8 +++++
t/t9503/test_cache_interface.pl | 63 +++++++++++++++++++++++++++++++++++++-
3 files changed, 86 insertions(+), 8 deletions(-)
commit 19911970b8a811a6382e39a10b071bff1dd4bd70
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:20:46 2010 +0100
gitweb/cache.pm - Regenerate (refresh) cache in background
This commit removes assymetry in serving stale data (if it exists)
when regenerating cache in GitwebCache::SimpleFileCache. The process
that acquired exclusive (writers) lock, and is therefore selected to
be the one that (re)generates data to fill the cache, can now generate
data in background, while serving stale data.
This feature can be enabled or disabled on demand via 'background_cache'
cache parameter. It is turned on by default.
To be implemented (from original patch by J.H.):
* server-side progress indicator when waiting for filling cache,
which in turn requires separating situations (like snapshots and
other non-HTML responses) where we should not show 'please wait'
message
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 36 +++++++++++++++++++++++++++++-------
gitweb/gitweb.perl | 9 +++++++++
t/t9503/test_cache_interface.pl | 14 ++++++++------
3 files changed, 46 insertions(+), 13 deletions(-)
commit ce97bb5bc1660f6d5c9b9be68c556ac94097978c
Author: Jakub Narebski <jnareb@gmail.com>
Date: Sun Feb 7 13:21:10 2010 +0100
gitweb: Show appropriate "Generating..." page when regenerating cache
When there exist stale/expired (but not too stale) version of
(re)generated page in cache, gitweb returns stale version (and updates
cache in background, assuming 'background_cache' is set to true value).
When there is no stale version suitable to serve the client, currently
we have to wait for the data to be generated in full before showing it.
Add to GitwebCache::SimpleFileCache, via 'generating_info' callback,
the ability to show user some activity indicator / progress bar, to
show that we are working on generating data.
Gitweb itself uses "Generating..." page as activity indicator, which
redirects (via <meta http-equiv="Refresh" ...>) to refreshed version
of the page after the cache is filled (via trick of not closing page
and therefore not closing connection till data is available in cache,
checked by getting shared/readers lock on lockfile for cache entry).
The git_generating_data_html() subroutine, which is used by gitweb
to implement this feature, is highly configurable: you can choose
initial delay, frequency of writing some data so that connection
won't get closed, and maximum time to wait for data in "Generating..."
page (see %generating_options hash).
Currently git_generating_data_html() contains hardcoded "whitelist" of
actions for which such HTML "Generating..." page makes sense.
This implements final feature from the original gitweb output caching
patch by J.H.
Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
gitweb/cache.pm | 23 +++++-
gitweb/gitweb.perl | 154 ++++++++++++++++++++++++++++++++++++++-
t/t9503/test_cache_interface.pl | 45 +++++++++++
3 files changed, 216 insertions(+), 6 deletions(-)
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing for caching)
[not found] ` <0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557.1264198194.git.jnareb@gmail.com>
2010-01-23 0:48 ` [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing for caching) Jakub Narebski
@ 2010-02-07 21:32 ` Jakub Narebski
1 sibling, 0 replies; 60+ messages in thread
From: Jakub Narebski @ 2010-02-07 21:32 UTC (permalink / raw)
To: git
On Sat, 23 Feb 2010, Jakub Narebski wrote:
> This means replacing
>
> print <something>;
> by
> print {$out} <something>;
>
> and
>
> binmode STDOUT, <layer>;
> by
> binmode $out, <layer>;
[...]
> This patch was written _before_ comment from Junio that it would be
> better to simply use
> print $out <something>;
> and do not try to be too clever.
Actually 'print FILEHANDLE LIST' form is an example of indirect object
syntax in Perl (which for filehandles was present before there were
objects in Perl). In indirect object notation the "first argument"
must be a name, a scalar variable, or a block (like {$out}). If instead
of $out there would be something more complicated, we would have to
use block form ('print {$out[$i]} <sth>', for example).
--
Jakub Narebski
Poland
^ permalink raw reply [flat|nested] 60+ messages in thread
end of thread, other threads:[~2010-02-07 21:32 UTC | newest]
Thread overview: 60+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-01-14 1:22 [PATCH 0/9] Gitweb caching v5 John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 1/9] gitweb: Load checking John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information John 'Warthog9' Hawley
2010-01-14 1:22 ` [PATCH 3/9] gitweb: Add option to force version match John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 4/9] gitweb: Makefile improvements John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 5/9] gitweb: add a get function to compliment print_local_time John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 6/9] gitweb: add a get function to compliment print_sort_th John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle John 'Warthog9' Hawley
2010-01-14 1:23 ` [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org) John 'Warthog9' Hawley
2010-01-16 2:48 ` Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 00/10] gitweb: Simple file based output caching Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1) Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set) output caching (WIP) Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP) Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP) Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd' problem (WIP) Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for filling cache (WIP) Jakub Narebski
2010-01-23 0:27 ` [RFC PATCH 10/10] gitweb: Show appropriate "Generating..." page when regenerating " Jakub Narebski
2010-01-24 22:24 ` Petr Baudis
2010-01-25 0:03 ` Jakub Narebski
2010-01-25 1:17 ` Jakub Narebski
2010-01-25 11:46 ` Jakub Narebski
2010-01-25 13:02 ` Petr Baudis
2010-01-25 13:48 ` Jakub Narebski
2010-01-25 13:56 ` Petr Baudis
2010-01-25 20:32 ` J.H.
2010-01-26 1:49 ` Jakub Narebski
2010-01-28 17:39 ` Petr Baudis
2010-01-31 11:58 ` Jakub Narebski
2010-01-25 20:58 ` Jakub Narebski
2010-01-25 20:41 ` J.H.
2010-01-26 2:30 ` Jakub Narebski
2010-01-23 19:55 ` [RFC PATCH 00/10] gitweb: Simple file based output caching J.H.
2010-01-24 13:54 ` [RFC PATCH 11/10] gitweb: Ajax-y "Generating..." page when regenerating cache (WIP) Jakub Narebski
2010-02-06 0:51 ` [RFC PATCH 00/10] gitweb: Simple file based output caching J.H.
2010-02-06 23:56 ` Jakub Narebski
2010-02-07 12:35 ` Jakub Narebski
[not found] ` <0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557.1264198194.git.jnareb@gmail.com>
2010-01-23 0:48 ` [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing for caching) Jakub Narebski
2010-02-07 21:32 ` Jakub Narebski
2010-01-16 0:43 ` [PATCH 8/9] gitweb: Convert output to using indirect file handle Jakub Narebski
2010-01-16 0:58 ` Junio C Hamano
2010-01-16 1:14 ` Jakub Narebski
2010-01-16 1:41 ` Junio C Hamano
2010-01-24 22:14 ` Petr Baudis
2010-01-25 1:47 ` Jakub Narebski
2010-01-25 20:48 ` J.H.
2010-01-25 21:48 ` Jakub Narebski
2010-01-15 23:49 ` [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header Jakub Narebski
2010-01-23 11:13 ` [PATCH 5/9] gitweb: add a get function to compliment print_local_time Jakub Narebski
2010-01-15 23:36 ` [PATCH 3/9] gitweb: Add option to force version match Jakub Narebski
2010-01-24 21:59 ` Petr Baudis
2010-01-24 23:17 ` Jakub Narebski
2010-01-15 22:40 ` [PATCH 2/9] gitweb: change die_error to take "extra" argument for extended die information Jakub Narebski
2010-01-15 22:30 ` [PATCH 1/9] gitweb: Load checking Jakub Narebski
2010-01-15 1:40 ` [PATCH 0/9] Gitweb caching v5 Jakub Narebski
2010-01-15 4:29 ` J.H.
2010-01-15 10:28 ` Jakub Narebski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.