Git documentation at kernel.org

Git documentation at kernel.org

[Petr Onderka]

Hi,

since the hacking of kernel.org, the online version of git
documentation [1] is not available. I realize that I can use local
version of the documentation and there is also at least one mirror
[2]. But I think it's very useful to have an "official" version of the
documentation online. And, more importantly, there is now lots of dead
links all over the Internet to the kernel.org version of the
documentation.

Can someone with the ability to do so restore the documentation at the
old location?

Thanks.

Petr Onderka

[1]: http://www.kernel.org/pub/software/scm/git/docs/git.html
[2]: http://schacon.github.com/git/git.html

Re: Git documentation at kernel.org

[Clemens Buchacher]

Hi,

Please restore access to the following files when possible. Some sites
are referencing those, including kernel.org itself:

 http://www.kernel.org/pub/software/scm/git/docs/git.html
 and references therein

 http://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html
 referenced by
 https://git.wiki.kernel.org/articles/g/i/t/GitFaq_ebc3.html#How_do_I_clone_a_subdirectory.3F

Also, it would be great if the git wiki could be made editable again.

Thank you for your consideration.

Regards,
Clemens

Re: Git documentation at kernel.org

[Junio C Hamano]

Clemens Buchacher <drizzd@aon.at> writes:

> Please restore access to the following files when possible. Some sites
> are referencing those, including kernel.org itself:
>
>  http://www.kernel.org/pub/software/scm/git/docs/git.html

The pages reachable from this used to be living documents in that every
time the 'master' branch was updated at k.org, automatically a server side
hook script generated a new set of HTML pages and updated them.

My understanding is that we do not want to run such random server side
hooks at k.org, so it no longer can be a living document anymore.

It might be a workable short term workaround to redirect

    http://www.kernel.org/pub/software/scm/git/docs/$anything

to

    http://schacon.github.com/git/$anything

although that would not give you an access to the list of documentations
for older releases, e.g.

    http://www.kernel.org/pub/software/scm/git/docs/v1.6.0/git.html

> Also, it would be great if the git wiki could be made editable again.

Amen.

Re: Git documentation at kernel.org

[Neal Kreitzinger]

On 2/7/2012 6:28 AM, Petr Onderka wrote:

> since the hacking of kernel.org, the online version of git
> documentation [1] is not available. I realize that I can use local
> version of the documentation and there is also at least one mirror
> [2]. But I think it's very useful to have an "official" version of the
> documentation online. And, more importantly, there is now lots of dead
> links all over the Internet to the kernel.org version of the
> documentation.
>
> Can someone with the ability to do so restore the documentation at the
> old location?
>
Additional info: 
http://article.gmane.org/gmane.comp.version-control.git/185302

v/r,
neal

Re: Git documentation at kernel.org

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Clemens Buchacher <drizzd@aon.at> writes:
>
>> Please restore access to the following files when possible. Some sites
>> are referencing those, including kernel.org itself:
>>
>>  http://www.kernel.org/pub/software/scm/git/docs/git.html
>
> The pages reachable from this used to be living documents in that every
> time the 'master' branch was updated at k.org, automatically a server side
> hook script generated a new set of HTML pages and updated them.

Is it possible to have the static HTML uploaded from another machine,
not necessarily for each push, but e.g. for every release?

I don't think anyone cares about having the very latest documentation
there, but it would still be great to have an official place to point to
when writing documentation on the web about such or such command.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Git documentation at kernel.org

[Theodore Tso]

On Feb 10, 2012, at 11:59 AM, Matthieu Moy wrote:
> 
> Is it possible to have the static HTML uploaded from another machine,
> not necessarily for each push, but e.g. for every release?
> 
> I don't think anyone cares about having the very latest documentation
> there, but it would still be great to have an official place to point to
> when writing documentation on the web about such or such command.

I think that's a great idea.

I used to use that service quite a lot, so I'd be willing to push the tar balls
to kernel.org, since I have PGP key set up for kernel.org uploads (so do
other people, and if someone else wants to do it, I'm happy to let them get
all the glory :-)   Most of the infrastructure to do this has been implemented,
except for the part where the tar ball gets unpacked in the correct directory.

This would satisfy the security concerns, and it wouldn't be hard, but it would
require some implementation work.   Anyone have some perl hacking time to
take a look at: 

      git://git.kernel.org/pub/scm/utils/kup/kup.git

… and add a "UNPACK pathanme" to the kup-server file, and work with the
sysadmins at kernel.org to get it reviewed and accepted?

-- Ted

Re: Git documentation at kernel.org

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

On Fri, 2012-02-10 at 13:00 -0500, Theodore Tso wrote:
> This would satisfy the security concerns, and it wouldn't be hard, but it would
> require some implementation work.   Anyone have some perl hacking time to
> take a look at: 
> 
>       git://git.kernel.org/pub/scm/utils/kup/kup.git
> 
> … and add a "UNPACK pathanme" to the kup-server file, and work with the
> sysadmins at kernel.org to get it reviewed and accepted?

I have a few comments off the top of my head:

     1. "kup rm" will need to be modified, as it currently only allows
        deleting things that have a matching signature. The alternative
        is for UNPACK to create a foo.tar.manifest file that will be
        consulted upon "kup rm" to clean up any unpacked contents upon
        the deletion of the source archive. Note, that there are many,
        many gotchas with this solution -- e.g. .manifest should
        probably contain checksums, too, as there are bound to be
        conditions when two tarballs reference the same files, and you
        want to make sure that you delete files matching the contents of
        the old tarball, not the newer one, etc.
     2. I would suggest that UNPACK ignores any directory structure in
        the archive, and only copies over files matching a restricted
        set of extensions (.html, .txt, .jpg, .png) into the same dir as
        the original tarball. Basically, untar into a temporary
        directory, then find any files matching the above set of
        extensions, copy them into another temporary location, force
        permissions to 0644, and then move them into the final "live"
        location in the same dir with the tarball (with the
        corresponding .manifest, if that solution used). There should be
        logic to make sure that we never overwrite any files that have a
        matching .sign file.
     3. There should be some support to ensure that the unpack process
        is terminated if unpacked content size reaches a certain limit,
        or if it is taking too long to complete.

Best regards,
-- 
Konstantin Ryabitsev
Systems Administrator, Kernel.org
Montréal, Québec

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 665 bytes --]

Re: Git documentation at kernel.org

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

> Junio C Hamano <gitster@pobox.com> writes:
>
>> Clemens Buchacher <drizzd@aon.at> writes:
>>
>>> Please restore access to the following files when possible. Some sites
>>> are referencing those, including kernel.org itself:
>>>
>>>  http://www.kernel.org/pub/software/scm/git/docs/git.html
>>
>> The pages reachable from this used to be living documents in that every
>> time the 'master' branch was updated at k.org, automatically a server side
>> hook script generated a new set of HTML pages and updated them.
>
> Is it possible to have the static HTML uploaded from another machine,
> not necessarily for each push, but e.g. for every release?

It would probably be possible, but I do not have that much time and
patience to sign 600+ files in the preformatted HTML tree one-by-one and
upload them using kup.

Re: Git documentation at kernel.org

[Ted Ts'o]

How about this as something *way* simpler?  Define a way of marking
the top of a particular directory hierarchy as a tree.  Then the
*only* way of updating that tree is all or nothing.  That is, someone
submits a signed tarball; then after the signed tarball has its
signature checked, it gets unpacked into a dir.new, and then we rename
dir to dir.old, rename dir.new to dir, and then dir.old gets removed.

That way there's no conflicts between directories that are managed via
the kup-servers PUT and DELETE commands, and those where they get
uploaded as a single tarball to create or replace a specific directory
hierarcy, or which can be deleted only as a entire directory hierarcy.

What do you think?

						- Ted

Re: Git documentation at kernel.org

[Jeff King]

On Fri, Feb 10, 2012 at 01:55:54PM -0500, Konstantin Ryabitsev wrote:

> I have a few comments off the top of my head:
> 
>      1. "kup rm" will need to be modified, as it currently only allows
>         deleting things that have a matching signature. The alternative
>         is for UNPACK to create a foo.tar.manifest file that will be
>         consulted upon "kup rm" to clean up any unpacked contents upon
>         the deletion of the source archive. Note, that there are many,
>         many gotchas with this solution -- e.g. .manifest should
>         probably contain checksums, too, as there are bound to be
>         conditions when two tarballs reference the same files, and you
>         want to make sure that you delete files matching the contents of
>         the old tarball, not the newer one, etc.

For this particular use case, I don't know if that would be necessary.
According to Junio, previously:

  The k.org site kept these files under /pub/software/scm/git/docs/. The
  in-development "master" version of pages were placed directly
  underneath that directory, and the documentation pages for older
  versions were kept in vX.Y.Z subdirectory of that directory.

If we tweak that slightly to "all versions are kept in vX.Y.Z
subdirectory, and the root version is simply a symlink or redirect to
the latest vX.Y.Z", then there is no deletion required. The pusher is
always adding new versions, and updating a link[1].

But even if it would be sufficient for this use case, kup developers
may not want such a half-implemented scheme in their protocol.

-Peff

[1] There is a slight complication that the subdirectories live _inside_
    of the root directory, so it is not implementable with a single
    symlink.  You could get around that with a few clever http redirects.

Re: Git documentation at kernel.org

[Jeff King]

On Thu, Feb 09, 2012 at 04:23:01PM -0800, Junio C Hamano wrote:

> It might be a workable short term workaround to redirect
> 
>     http://www.kernel.org/pub/software/scm/git/docs/$anything
> 
> to
> 
>     http://schacon.github.com/git/$anything
> 
> although that would not give you an access to the list of documentations
> for older releases, e.g.
> 
>     http://www.kernel.org/pub/software/scm/git/docs/v1.6.0/git.html

If there is interest in this, we would be happy to host the
documentation. Let me know if that is the case, and we can give it a
much better URL than schacon.github.com. However, I tend to think that
since the project is hosted[1] at kernel.org, the official documentation
site should be there as well.

-Peff

[1] Of course, git being git, it is not really hosted _anywhere_ in
    particular. But convention thus far has said that the kernel.org
    repository is the official one.

Re: Git documentation at kernel.org

[Junio C Hamano]

Ted Ts'o <tytso@mit.edu> writes:

> How about this as something *way* simpler?  Define a way of marking
> the top of a particular directory hierarchy as a tree.  Then the
> *only* way of updating that tree is all or nothing.  That is, someone
> submits a signed tarball; then after the signed tarball has its
> signature checked, it gets unpacked into a dir.new, and then we rename
> dir to dir.old, rename dir.new to dir, and then dir.old gets removed.
>
> That way there's no conflicts between directories that are managed via
> the kup-servers PUT and DELETE commands, and those where they get
> uploaded as a single tarball to create or replace a specific directory
> hierarcy, or which can be deleted only as a entire directory hierarcy.
>
> What do you think?

That would not work very well without changing the historical directory
structure (which I think was the point of this discussion "please keep
these stale links alive").

The toplevel index.html in the pub/software/scm/git/docs/ directory and
its pointees were the set of docs for the latest version, and older
versions were rooted at pub/software/scm/git/docs/vX.Y.Z/.  Links that
point at software/scm/git/docs/git-cat-file.html still need to work, and
the path needs to be updatable without having to include the preformatted
documentation for all the historical versions in the same tarball.

Re: Git documentation at kernel.org

[Ted Ts'o]

On Fri, Feb 10, 2012 at 12:18:35PM -0800, Junio C Hamano wrote:
> That would not work very well without changing the historical directory
> structure (which I think was the point of this discussion "please keep
> these stale links alive").
> 
> The toplevel index.html in the pub/software/scm/git/docs/ directory and
> its pointees were the set of docs for the latest version, and older
> versions were rooted at pub/software/scm/git/docs/vX.Y.Z/.  Links that
> point at software/scm/git/docs/git-cat-file.html still need to work, and
> the path needs to be updatable without having to include the preformatted
> documentation for all the historical versions in the same tarball.

Hmm... good point.  That does make it hard.  I could imagine making it
work by having separate hierarchies, and then using apache rewrite
rules so that anything that doesn't begin with vX.Y.Z in the top level
of software/scm/git/docs/* gets redirected to LATEST/*, where LATEST is
a symlink that is managed via kup.

I don't know if the k.org folks would consider that acceptable, though.

  	     	    	  	      	   - Ted

Re: Git documentation at kernel.org

[Junio C Hamano]

Ted Ts'o <tytso@mit.edu> writes:

> Hmm... good point.  That does make it hard.  I could imagine making it
> work by having separate hierarchies, and then using apache rewrite
> rules so that anything that doesn't begin with vX.Y.Z in the top level
> of software/scm/git/docs/* gets redirected to LATEST/*, where LATEST is
> a symlink that is managed via kup.

We could move vX.Y.Zs out of scm/git/docs/ hierarchy.  The existing links
from external sites rarely point at a documentation page of a specific
version, I suspect.

People can be trained to look at scm/git/old-docs/vX.Y.Z when they want to
see how older command set looked like, even those who know that in olden
days they would have consulted scm/git/docs/vX.Y.Z for that information;
they are much less of a problem than existing pages whose links want to
stay working.

Re: Git documentation at kernel.org

[Matthieu Moy]

Jeff King <peff@peff.net> writes:

> If there is interest in this, we would be happy to host the
> documentation. Let me know if that is the case, and we can give it a
> much better URL than schacon.github.com. However, I tend to think that
> since the project is hosted[1] at kernel.org, the official documentation
> site should be there as well.

kernel.org is probably the most "official" place for developers, but for
Git users, http://git-scm.com/ is most likely the best entry point. If
it were not for historical reasons, I think http://git-scm.com/docs/ or
so would be the most natural URL to host official docs.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Git documentation at kernel.org

[Jeff King]

On Sun, Feb 12, 2012 at 11:04:23PM +0100, Matthieu Moy wrote:

> Jeff King <peff@peff.net> writes:
> 
> > If there is interest in this, we would be happy to host the
> > documentation. Let me know if that is the case, and we can give it a
> > much better URL than schacon.github.com. However, I tend to think that
> > since the project is hosted[1] at kernel.org, the official documentation
> > site should be there as well.
> 
> kernel.org is probably the most "official" place for developers, but for
> Git users, http://git-scm.com/ is most likely the best entry point. If
> it were not for historical reasons, I think http://git-scm.com/docs/ or
> so would be the most natural URL to host official docs.

Good point. That is probably the best place to host it.

As far as historical reasons, perhaps the right answer is to put the
documentation where it makes sense to go _now_, and ask kernel.org to
issue http redirects for http://kernel.org/pub/software/scm/git/docs.

-Peff

Re: Git documentation at kernel.org

[Scott Chacon]

Hey,

On Sun, Feb 12, 2012 at 2:25 PM, Jeff King <peff@peff.net> wrote:
>> kernel.org is probably the most "official" place for developers, but for
>> Git users, http://git-scm.com/ is most likely the best entry point. If
>> it were not for historical reasons, I think http://git-scm.com/docs/ or
>> so would be the most natural URL to host official docs.
>
> Good point. That is probably the best place to host it.
>
> As far as historical reasons, perhaps the right answer is to put the
> documentation where it makes sense to go _now_, and ask kernel.org to
> issue http redirects for http://kernel.org/pub/software/scm/git/docs.

I would be happy to set this up.  I'm currently in the process of
revamping the website and this is one of the things I'm planning on
doing anyways - not just hosting the generated docs, but also making
them searchable and whatnot.

Actually, as long as I'm on this, what do people think about git-scm
hosting the wiki as well?  As far as I can tell, it was down for
months and now it's back in some sort of weird read-only state.  If I
imported everything into a different wiki and hosted it on git-scm
would that be acceptable?

Also, something that I realized I am not willing to maintain any more
is the Git Community Book. It was an experiment at reorganizing some
of the docs, but instead I spent my time on Pro Git, which is CC
licensed.  Would anyone object to me removing the community book from
the git-scm site and more tightly integrating the Pro Git content?
It's more up to date and better content, I feel - I would rather have
one book to maintain than two.  However, since it is a commercial
product (albeit a Creative Commons licensed one), I wasn't sure if
people would have an issue with it.

Scott

Re: Git documentation at kernel.org

[Jeff King]

On Sun, Feb 12, 2012 at 03:04:59PM -0800, Scott Chacon wrote:

> > Good point. That is probably the best place to host it.
> >
> > As far as historical reasons, perhaps the right answer is to put the
> > documentation where it makes sense to go _now_, and ask kernel.org to
> > issue http redirects for http://kernel.org/pub/software/scm/git/docs.
> 
> I would be happy to set this up.  I'm currently in the process of
> revamping the website and this is one of the things I'm planning on
> doing anyways - not just hosting the generated docs, but also making
> them searchable and whatnot.

That sounds great to me. I'd like to be link-compatible with the old
kernel.org docs section (even if through redirects) so that old links
work (assuming kernel.org gives us a wholesale redirect).  Which means
importing all of the docs for released versions. I don't know if the old
kernel.org doc tree was saved anywhere, but if I understand correctly,
they are identical to what's in the "git-htmldocs" repository (which I
_thought_ Junio wasn't going to keep updating, but it seems pretty up to
date).

> Actually, as long as I'm on this, what do people think about git-scm
> hosting the wiki as well?  As far as I can tell, it was down for
> months and now it's back in some sort of weird read-only state.  If I
> imported everything into a different wiki and hosted it on git-scm
> would that be acceptable?

I'd really love it if the wiki was converted to something that was
git-backed. But I suspect some people might complain about switching off
of mediawiki. IIRC, gollum supports some mediawiki syntax, but I don't
know how much conversion work there would be.

> Also, something that I realized I am not willing to maintain any more
> is the Git Community Book. It was an experiment at reorganizing some
> of the docs, but instead I spent my time on Pro Git, which is CC
> licensed.  Would anyone object to me removing the community book from
> the git-scm site and more tightly integrating the Pro Git content?
> It's more up to date and better content, I feel - I would rather have
> one book to maintain than two.  However, since it is a commercial
> product (albeit a Creative Commons licensed one), I wasn't sure if
> people would have an issue with it.

I can't remember anybody mentioning the Git Community Book here in the
past few years. New users typically come with a "I read this in Pro Git
and I don't understand..." question, and experienced users recommend or
link to Pro Git. So I think the world would be a less confusing place
with just the one source.

-Peff

Re: Git documentation at kernel.org

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> That sounds great to me. I'd like to be link-compatible with the old
> kernel.org docs section (even if through redirects) so that old links
> work (assuming kernel.org gives us a wholesale redirect).  Which means
> importing all of the docs for released versions. I don't know if the old
> kernel.org doc tree was saved anywhere, but if I understand correctly,
> they are identical to what's in the "git-htmldocs" repository (which I
> _thought_ Junio wasn't going to keep updating, but it seems pretty up to
> date).

I have been updating htmldocs/manpages repositories on "unless I forget"
basis every time the public 'master' gets updated, so they are updated as
frequently as they used to back when they were autogenerated, "unless I
forget".

But the contents of htmldocs does _not_ match what used to be at k.org in
that its git.html does not have links to documentation pages for older
releases, iow, formatted without "stalenotes" defined.

This is because formatting with "stalenotes" needs to make an assumption
on the filesystem layout that I cannot enforce to the users of htmldocs
repository. They will get one tarball for one version, and it is up to
them where they extract these tarballs.  They need to extract the tarball
of an older release vX.Y.Z in vX.Y.Z subdirectory next to the git.html of
the latest living document to match the layout, but otherwise the links
created by "stalenotes" will become dangling.

Re: Git documentation at kernel.org

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On Sun, 2012-02-12 at 17:25 -0500, Jeff King wrote:
> As far as historical reasons, perhaps the right answer is to put the
> documentation where it makes sense to go _now_, and ask kernel.org to
> issue http redirects for http://kernel.org/pub/software/scm/git/docs. 

I think that should be fine, unless John objects. The easiest would be
to preserve the same directory structure, so we do a dir-level redirect
instead of creating one-off redirects for each page.

Best,
-- 
Konstantin Ryabitsev
Systems Administrator, Kernel.org
Montréal, Québec

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 665 bytes --]

Re: Git documentation at kernel.org

[Jeff King]

On Sun, Feb 12, 2012 at 07:23:19PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > That sounds great to me. I'd like to be link-compatible with the old
> > kernel.org docs section (even if through redirects) so that old links
> > work (assuming kernel.org gives us a wholesale redirect).  Which means
> > importing all of the docs for released versions. I don't know if the old
> > kernel.org doc tree was saved anywhere, but if I understand correctly,
> > they are identical to what's in the "git-htmldocs" repository (which I
> > _thought_ Junio wasn't going to keep updating, but it seems pretty up to
> > date).
> [...]
> But the contents of htmldocs does _not_ match what used to be at k.org in
> that its git.html does not have links to documentation pages for older
> releases, iow, formatted without "stalenotes" defined.

Ah, that makes sense. We might have to just rebuild the old versions
with stalenotes, then (our doc toolchain is so finicky that I worry
about minor incompatibilities in building old versions with a newer
toolchain, but it is probably good enough).

-Peff