Pushing a git repository to a new server

Pushing a git repository to a new server

[Ethan Reesor]

I'm looking to make a command to push a git repo to a new server. The
way I just did it is as follows:

localhost> git clone --bare /path/to/MyRepo /path/to/tmpdir/MyRepo.git
localhost> tar xz /path/to/tmpdir/MyRepo.git | ssh myuser@remotehost
tar cz \~/      # If I don't escape '~', my local machine expands it
localhost> ssh myuser@remotehost
remotehost> sudo chown -R git:git MyRepo.git

The reason I had to use my user is the git user's shell is git-prompt
and ~git/git-shell-commands is empty. I have repos set up using
'git@remotehost:MyOtherRepo.git' as the remote and everything works.

How do I make a git command that can talk to the server using
git-prompt like the other commands do?

--
Ethan Reesor

Re: Pushing a git repository to a new server

[Konstantin Khomoutov]

On Sun, Feb 10, 2013 at 04:00:56PM -0500, Ethan Reesor wrote:

> I'm looking to make a command to push a git repo to a new server. The
> way I just did it is as follows:
> 
> localhost> git clone --bare /path/to/MyRepo /path/to/tmpdir/MyRepo.git
> localhost> tar xz /path/to/tmpdir/MyRepo.git | ssh myuser@remotehost
> tar cz \~/      # If I don't escape '~', my local machine expands it
> localhost> ssh myuser@remotehost
> remotehost> sudo chown -R git:git MyRepo.git

What's wrong with
$ ssh myuser@remotehost 'mkdir /path/to/MyRepo.git; cd $_; git init --bare'
$ git push --all git@remotehost:MyOtherRepo.git
?

> The reason I had to use my user is the git user's shell is git-prompt

There's no such thing as git-prompt.  The restricted login shell for
SSH-only access typically used for such a "virtual" Git user is
git-shell.

> and ~git/git-shell-commands is empty. I have repos set up using
> 'git@remotehost:MyOtherRepo.git' as the remote and everything works.
> 
> How do I make a git command that can talk to the server using
> git-prompt like the other commands do?

It's not really clear what do you want to achieve.
The reason the git-shell shell is *restricted* (read its manual page)
is to shrink the surface of possible attacks in the case the shell
account used for accessing Git repos over SSH is compromized (the key or
password stolen, for instance).  This is achieved by only allowing
commands like git-upload-pack etc in the shell (no general file
manipulation commands etc).  So what creating "git command that can
talk to the server using git-prompt ..." would really buy you?

I think the way to go is to start using gitolite [1] or implement by
hand a subset of what it does (a custom login shell which is allowed to
do certain things in a special area of the filesystem designated to keep
Git repositories) or just set up a special account on the server
("git-admin", for instance) which would have a regular login shell set
for it and would be in the same group as the user "git" (or even have
the same UID) so that they could share the files they create (subject to
active umasks of processes run as both users though).

1. https://github.com/sitaramc/gitolite

Re: Pushing a git repository to a new server

[Ethan Reesor]

On Mon, Feb 11, 2013 at 2:50 AM, Konstantin Khomoutov
<kostix+git@007spb.ru> wrote:
> What's wrong with
> $ ssh myuser@remotehost 'mkdir /path/to/MyRepo.git; cd $_; git init --bare'
> $ git push --all git@remotehost:MyOtherRepo.git
> ?

Nothing, I just wanted to make myself a command to do that for me.

>> The reason I had to use my user is the git user's shell is git-prompt
>
> There's no such thing as git-prompt.  The restricted login shell for
> SSH-only access typically used for such a "virtual" Git user is
> git-shell.

Sorry, git-prompt is something I made for myself. I meant git-shell.

> It's not really clear what do you want to achieve.
> The reason the git-shell shell is *restricted* (read its manual page)
> is to shrink the surface of possible attacks in the case the shell
> account used for accessing Git repos over SSH is compromized (the key or
> password stolen, for instance).  This is achieved by only allowing
> commands like git-upload-pack etc in the shell (no general file
> manipulation commands etc).  So what creating "git command that can
> talk to the server using git-prompt ..." would really buy you?

I want to create a git-command that 1) creates a bare version of the
current repo, 2) and uploads it to the specified path on my server
(using tar, but that's not the point).

My problem is that I have no idea how things like git-push works via a
user with git-shell. Can you only run certain git commands, like
git-upload-pack? Because I tried running 'ssh git@server git status'
and that failed.

> I think the way to go is to start using gitolite [1] or implement by
> hand a subset of what it does (a custom login shell which is allowed to
> do certain things in a special area of the filesystem designated to keep
> Git repositories) or just set up a special account on the server
> ("git-admin", for instance) which would have a regular login shell set
> for it and would be in the same group as the user "git" (or even have
> the same UID) so that they could share the files they create (subject to
> active umasks of processes run as both users though).

I thought about the secondary user idea. I decided that trying to make
my own command would be more fun.


-- 
Ethan Reesor (Gmail)

Re: Pushing a git repository to a new server

[Konstantin Khomoutov]

On Mon, 11 Feb 2013 02:57:51 -0500
Ethan Reesor <firelizzard@gmail.com> wrote:

[...]
> I want to create a git-command that 1) creates a bare version of the
> current repo, 2) and uploads it to the specified path on my server
> (using tar, but that's not the point).

Thanks, it's now a bit more clear.

> My problem is that I have no idea how things like git-push works via a
> user with git-shell. Can you only run certain git commands, like
> git-upload-pack?
[...]

Precisely so.  With additional twist that you can create (or link)
other commands under ~/git-shell-commands, and these will be available
as well.

OK, here's the sketch.
On the server, in the home directory of your "git" user, you create a
wrapper around git-receive-pack, like this:

# mkdir ~git/git-shell-commands
# cat >~git/git-shell-commands/git-receive-new-repo
#!/bin/sh

set -e -u

if [ $# -ne 1 ]; then
        echo 'Missing required argument: <directory>' >&2
        exit 1
fi

mkdir "$1" && git init --quiet --bare "$1" && git-receive-pack "$1"
^D
# chmod +x $_

Then, on the client side, to push a new repo, you just do

$ git push --receive-pack=git-receive-new-repo --all git@server:repo.git

This will make `git push` to spawn not just `git receive-pack <dir>` as
it usually does but your wrapper, which would first create and
initialize a bare repository and then spawn `git receive-pack` on it
which would then communicate with the client side and receive
everything from it.

You could then create a client-side wrapper script or a Git alias for
such "creative pushing", like this:

$ git config --add --global alias.push-new-repo \
  'push --receive-pack=git-receive-new-repo --all'

So the whole client call is now reduced to

$ git push-new-repo git@server:repo.git

Re: Pushing a git repository to a new server

[Jeff King]

On Mon, Feb 11, 2013 at 02:57:51AM -0500, Ethan Reesor wrote:

> On Mon, Feb 11, 2013 at 2:50 AM, Konstantin Khomoutov
> <kostix+git@007spb.ru> wrote:
> > What's wrong with
> > $ ssh myuser@remotehost 'mkdir /path/to/MyRepo.git; cd $_; git init --bare'
> > $ git push --all git@remotehost:MyOtherRepo.git
> > ?
> 
> Nothing, I just wanted to make myself a command to do that for me.

We talked about this a long time ago. One problem is that it's
inherently unportable, as the procedure to make a repo is potentially
different on every server (and certainly that is the case between a
regular user running stock git and something like GitHub or Google Code;
I imagine even gitolite has some special procedures for creating repos,
too).

One proposal made in the previous discussion was to define a microformat
for repository administration commands. So that you could connect and
say "git admin-create-repo /path/to/MyRepo.git", and the server-provided
admin-create-repo command would take care of the details. Then stock git
could forward it to "git init --bare", GitHub could do the same and
create the necessary database records, etc.

And once that standardized method was in place, it would be easy to add
a "--create" option to "git push" to request an "admin-create-repo"
before pushing.

I still think that's a reasonable way forward, but nobody was interested
enough to start writing code for it.

-Peff

Re: Pushing a git repository to a new server

[Ethan Reesor]

On Mon, Feb 11, 2013 at 11:27 AM, Jeff King <peff@peff.net> wrote:
[...]
> We talked about this a long time ago. One problem is that it's
> inherently unportable, as the procedure to make a repo is potentially
> different on every server (and certainly that is the case between a
> regular user running stock git and something like GitHub or Google Code;
> I imagine even gitolite has some special procedures for creating repos,
> too).

I was more interested in creating something for my self rather than
making any changes to the mainstream git.

Re: Pushing a git repository to a new server

[Ethan Reesor]

On Mon, Feb 11, 2013 at 7:45 AM, Konstantin Khomoutov
<kostix+git@007spb.ru> wrote:
[...]
> OK, here's the sketch.
> On the server, in the home directory of your "git" user, you create a
> wrapper around git-receive-pack, like this:
>
> # mkdir ~git/git-shell-commands
> # cat >~git/git-shell-commands/git-receive-new-repo
> #!/bin/sh
>
> set -e -u
>
> if [ $# -ne 1 ]; then
>         echo 'Missing required argument: <directory>' >&2
>         exit 1
> fi
>
> mkdir "$1" && git init --quiet --bare "$1" && git-receive-pack "$1"
> ^D
> # chmod +x $_
>
> Then, on the client side, to push a new repo, you just do
>
> $ git push --receive-pack=git-receive-new-repo --all git@server:repo.git
>
> This will make `git push` to spawn not just `git receive-pack <dir>` as
> it usually does but your wrapper, which would first create and
> initialize a bare repository and then spawn `git receive-pack` on it
> which would then communicate with the client side and receive
> everything from it.
>
> You could then create a client-side wrapper script or a Git alias for
> such "creative pushing", like this:
>
> $ git config --add --global alias.push-new-repo \
>   'push --receive-pack=git-receive-new-repo --all'
>
> So the whole client call is now reduced to
>
> $ git push-new-repo git@server:repo.git

Thanks, that's what I was going for.

Re: Pushing a git repository to a new server

[Michael J Gruber]

Jeff King venit, vidit, dixit 11.02.2013 17:27:
> On Mon, Feb 11, 2013 at 02:57:51AM -0500, Ethan Reesor wrote:
> 
>> On Mon, Feb 11, 2013 at 2:50 AM, Konstantin Khomoutov
>> <kostix+git@007spb.ru> wrote:
>>> What's wrong with
>>> $ ssh myuser@remotehost 'mkdir /path/to/MyRepo.git; cd $_; git init --bare'
>>> $ git push --all git@remotehost:MyOtherRepo.git
>>> ?
>>
>> Nothing, I just wanted to make myself a command to do that for me.
> 
> We talked about this a long time ago. One problem is that it's
> inherently unportable, as the procedure to make a repo is potentially
> different on every server (and certainly that is the case between a
> regular user running stock git and something like GitHub or Google Code;
> I imagine even gitolite has some special procedures for creating repos,
> too).
> 
> One proposal made in the previous discussion was to define a microformat
> for repository administration commands. So that you could connect and
> say "git admin-create-repo /path/to/MyRepo.git", and the server-provided
> admin-create-repo command would take care of the details. Then stock git
> could forward it to "git init --bare", GitHub could do the same and
> create the necessary database records, etc.
>
> And once that standardized method was in place, it would be easy to add
> a "--create" option to "git push" to request an "admin-create-repo"
> before pushing.
> 
> I still think that's a reasonable way forward, but nobody was interested
> enough to start writing code for it.
> 
> -Peff
> 

I'm not sure providers like GitHub would fancy an interface which allows
the programmatic creation of repos (giving a new meaning to "fork
bomb"). But I bet you know better ;-)

An alternative would be to teach git (the client) about repo types and
how to create them. After all, a repo URL "ssh://host/path" gives a
clear indication that "ssh host git init path" will create a repo. I'm
wondering whether it's more likely to convince providers (the server
side) or more is gained by covering the simpler cases client-side (our
side).

Michael

Re: Pushing a git repository to a new server

[Jeff King]

On Tue, Feb 12, 2013 at 12:28:53PM +0100, Michael J Gruber wrote:

> I'm not sure providers like GitHub would fancy an interface which allows
> the programmatic creation of repos (giving a new meaning to "fork
> bomb"). But I bet you know better ;-)

You can already do that:

  http://developer.github.com/v3/repos/#create

We rate-limit API requests, and I imagine we might do something similar
with create-over-git. But that is exactly the kind of implementation
detail that can go into a custom create-repo script.

> An alternative would be to teach git (the client) about repo types and
> how to create them. After all, a repo URL "ssh://host/path" gives a
> clear indication that "ssh host git init path" will create a repo.

But that's the point of a microformat. It _doesn't_ always work, because
the server may not allow arbitrary commands, or may have special
requirements on top of the "init". You can make the microformat be "git
init path", and servers can intercept calls to "git init" and translate
them into custom magic. But I think the world is a little simpler if we
define a new service type (alongside git-upload-pack, git-receive-pack,
etc), and let clients request it. Then it's clear what the client is
trying to do, it's easy for servers to hook into it, we can request it
over http, etc. And it can be extended over time to take more fields
(like repo description, etc).

I'm really not suggesting anything drastic. The wrapper case for ssh
would be as simple as a 3-line shell script which calls "git init" under
the hood, but it provides one level of indirection that makes
replacing/hooking it much simpler for servers. So the parts that are in
stock git would not be much work (most of the work would be on _calling_
it, but that is the same for adding a call to "git init").

I think the main reason the idea hasn't gone anywhere is that nobody
really cares _that_ much. People just don't create repositories that
often. I feel like this is one of those topics that comes up once a
year, and then nothing happens on it, because people just make their
repo manually and then stop caring about it.

Just my two cents, of course. :)

-Peff

Re: Pushing a git repository to a new server

[Michael J Gruber]

Jeff King venit, vidit, dixit 12.02.2013 21:42:
> On Tue, Feb 12, 2013 at 12:28:53PM +0100, Michael J Gruber wrote:
> 
>> I'm not sure providers like GitHub would fancy an interface which allows
>> the programmatic creation of repos (giving a new meaning to "fork
>> bomb"). But I bet you know better ;-)
> 
> You can already do that:
> 
>   http://developer.github.com/v3/repos/#create

Nice.

I knew you knew ;)

> We rate-limit API requests, and I imagine we might do something similar
> with create-over-git. But that is exactly the kind of implementation
> detail that can go into a custom create-repo script.
> 
>> An alternative would be to teach git (the client) about repo types and
>> how to create them. After all, a repo URL "ssh://host/path" gives a
>> clear indication that "ssh host git init path" will create a repo.
> 
> But that's the point of a microformat. It _doesn't_ always work, because
> the server may not allow arbitrary commands, or may have special
> requirements on top of the "init". You can make the microformat be "git
> init path", and servers can intercept calls to "git init" and translate
> them into custom magic. But I think the world is a little simpler if we
> define a new service type (alongside git-upload-pack, git-receive-pack,
> etc), and let clients request it. Then it's clear what the client is
> trying to do, it's easy for servers to hook into it, we can request it
> over http, etc. And it can be extended over time to take more fields
> (like repo description, etc).
> 
> I'm really not suggesting anything drastic. The wrapper case for ssh
> would be as simple as a 3-line shell script which calls "git init" under
> the hood, but it provides one level of indirection that makes
> replacing/hooking it much simpler for servers. So the parts that are in
> stock git would not be much work (most of the work would be on _calling_
> it, but that is the same for adding a call to "git init").
> 
> I think the main reason the idea hasn't gone anywhere is that nobody
> really cares _that_ much. People just don't create repositories that
> often. I feel like this is one of those topics that comes up once a
> year, and then nothing happens on it, because people just make their
> repo manually and then stop caring about it.
> 
> Just my two cents, of course. :)

Most repos are probably created by a local "git init" or "git clone", or
by clicking a button on a provider's web interface. The need for
git-create-repo seems to be restricted to:

- "command line folks" who use a provider for it's hosting service and
don't fancy a web interface for repo creation

- noobs who need to get their head wrapped around local, remote,
push/pull 'n' stuff...

For the server side git-create-repo to take off we would probably need
two things (besides the client support):

- Implement and ship a git-create-repo which makes this work for git
over ssh seamlessly. (Will take some to trickle down to servers in the
wild.)

- Get a large provider to offer this.

Gitosis/Gitolite are probably to follow easily. I'm beginning to like
idea ;)

Michael