recipe to use git for deployment

recipe to use git for deployment

[Neal Kreitzinger]

I'm trying to cook up automated mass deployment using git as the main 
ingredient.  Here's a recipe idea:

install git on deployed server under the home dir of the 'gittech' user 
and add that path only to the PATH of 'gittech'.

have the git repos on deployed server be under the 'gittech' home dir 
with worktree(s) assigned to deployment locations.  If people mess with 
the worktrees I will be able to tell with git status via 'gittech'.

I can ssh to deployed servers from a deployment server and do git fetch 
to get updates.

I can have a 'deployment manager' repo which contains script to apply 
patches from remote tracking branch to worktree.  It checks out commits 
at waypoints and at waypoint it executes needed file/data conversion 
programs, creates dir/file structures, etc.  If conversion validation 
program is successful it will continue to checkout next waypoint, etc, 
until worktree matches HEAD of remote tracking branch.

Data/files are commited to data tracking git repo at waypoints.  If 
waypoint validation fails then software version repo and data version 
repo are checked out to last good waypoint or back to startpoint.

Problem(1): cron patches.  The cron jobs portion I'm not sure of.  Need 
to learn more about cron.  Perhaps cron can run scripts that are 
symlinks to a worktree.  Cron worktree can have waypoints just like 
software and data worktrees.  The cron tabs, cron.daily, etc can be 
worktrees perhaps also.

Problem(2a): diskspace.  I think I will have to pull shallow clones to 
avoid growing history.  For deployment I only care about recent 
revisions.  Not sure if the above ideas will work with shallow clone.

Problem(2b): network latency.  Maybe I can have the git repo object 
store on the deployment server and the deployed git repo reference it to 
save diskpace on the deployed server.  That would probably be really slow.

I'm thinking the ingredients of this recipe could make a tasty git 
deployment.  Maybe different ways of mixing and cooking them and other 
ingredients can make a good dish.  Please let me know if you think this 
can be edible for security, palatable for reliability, and digestible by 
git.

v/r,
neal

Re: recipe to use git for deployment

[Marc Weber]

I always did "git pull" on the servers - but the history of my projects
is not that huge.. Thus I never cared.
Great: You can keep some server specific config settings and rebase
them - and you can do a fast git status to check whether file contents
have been modified (eg determine whether you've been hacked ..)

If you really care that much about history why not push a zip file using
git archive --format=zip and unpack that on the deployment server
instead?

For FTP access only this did a great job for small projects:
https://github.com/MarcWeber/git-ftp (-> git-ftp-minimal.sh)
It only copies changed files *and* checks whether they have been
modified on the server first (detecting work of others).
But that's probably not a good thing for automatic deployment.

Marc Weber

Re: recipe to use git for deployment

[Junio C Hamano]

Neal Kreitzinger <nkreitzinger@gmail.com> writes:

> I'm trying to cook up automated mass deployment using git as the main
> ingredient.

I think the standard answer to this is "don't do it".

You could of course add hooks to hack around it, but fundamentally Git was
designed to track source files (so the only sane behaviour for "checking
out" is to keep the timestamp of the working tree file to whatever the
underlying OS gives it when "checkout" happens, for example), so anything
that your "deployment strategy" wants conflicts with what the usual source
coutrol operation should do, you need to tweak around what Git does (the
same thing can be said for CVS or SVN, by the way).

Unless you do this with an understanding that you are merely using git "as
a better rsync" and take responsiblity of everything else that git-the-scm
does _not_ do for you, I suspect that you would be volunteering for a lot
of pain.

Re: recipe to use git for deployment

[Neal Kreitzinger]

On 5/4/2012 11:10 PM, Marc Weber wrote:
> I always did "git pull" on the servers - but the history of my projects
> is not that huge.. Thus I never cared.
> Great: You can keep some server specific config settings and rebase
> them -
So I could rebase apache virtual host conf files with generic template 
changes like symlink support, but keep the networking ip's.  Merge 
conflicts would be bad.  I guess that would be cause to reset --hard to 
a previous commit.

and you can do a fast git status to check whether file contents
> have been modified (eg determine whether you've been hacked ..)
>
With 'source is run' like html, css, etc. that would be a good way to 
enforce no-hacking-allowed support agreements.

> If you really care that much about history why not push a zip file using
> git archive --format=zip and unpack that on the deployment server
> instead?
>
That is what I'm currently doing.  By paying attention to tracked 
executable bits I don't even have to fix permissions on deployment.

The scriptable patching of a git repo (scripted checkout of remote 
tracking branch commits to worktree with inserted conversions and 
validations -- an interactive cherry-pick so to speak) allows for 
flexible conversions of varying patch levels.  Maybe git-sequencer will 
do this when it comes out.

Also, I could have a deployment tracking repo on one of my servers that 
remotes to all deployed servers and pulls to refresh my monitor view of 
all deployed revisions in gitk.

> For FTP access only this did a great job for small projects:
> https://github.com/MarcWeber/git-ftp (->  git-ftp-minimal.sh)
> It only copies changed files *and* checks whether they have been
> modified on the server first (detecting work of others).
> But that's probably not a good thing for automatic deployment.
>
I'll take a look to see what I can steal, uh I mean reuse.  Thanks!

v/r,
neal

Re: recipe to use git for deployment

[Marc Weber]

I've been deploying .php files this way only..

If you want to deploy full configuration of hosts you may also want to
learn about nixos.org (which handles restarting apache etc) or chef
like systemms:
http://www.rubyinside.com/chef-tasty-server-configuraiton-2162.html
using nixos you can easily automatically generate additional cron job
ensuring that permissions are correct etc.

Maybe talking about your exact use case will make others list more
options.

Marc Weber

Re: recipe to use git for deployment

[Neal Kreitzinger]

On 5/4/2012 11:10 PM, Marc Weber wrote:
>
> If you really care that much about history why not push a zip file
> using git archive --format=zip and unpack that on the deployment
> server instead?
>
I summon forth the ents to do my bidding
with one command to find them:  --format=tar
one command to transform them:  --prefix=deployed/path
one command to arm them:  config tar.umask (executable bits)
one command to gather them:  tar -A
and one command to bind them:  gzip

(We don't have the new tar with the transform option)

They are faithful and true and have won many battles, but I fear they 
may be too slow and lumbering for the gathering whirlwind of patches 
about to descend from above and legion of servers encircling them on all 
sides.

Interestingly, git-archive was not intended for deployment, but it does 
pretty good at it.

(Sorry, I haven't even read LOTR, but couldn't resist the opportunity to 
use the word 'ent' after reading about it in git-glossary ;)

v/r,
neal

Re: recipe to use git for deployment

[Ævar Arnfjörð Bjarmason]

On Sat, May 5, 2012 at 5:51 AM, Neal Kreitzinger <nkreitzinger@gmail.com> wrote:
> I'm trying to cook up automated mass deployment using git as the main
> ingredient.  Here's a recipe idea:

At work we use and have open sourced
https://github.com/git-deploy/git-deploy which we use for our
deployments.

It doesn't actually do the work of syncing out anything, we
effectively do that with distributed rsync.

Re: recipe to use git for deployment

[Sitaram Chamarty]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On Sat, May 5, 2012 at 11:00 AM, Neal Kreitzinger
<nkreitzinger@gmail.com> wrote:

> Interestingly, git-archive was not intended for deployment, but it does
> pretty good at it.

Except it doesn't delete files that should be deleted (the way 'git
checkout' does or rsync would if you gave it --delete).

On a lark I once wrote a very efficient way of dealing with this; see
attached.  It's probably won't work on anything but Linux and requires
bash, and it's only been minimally tested and I do not actually use
it, but someone might like it.

[-- Attachment #2: tar-checkout-f --]
[-- Type: application/octet-stream, Size: 705 bytes --]

#!/bin/bash

# WARNINGS:
#       THE DIRECTORY SPECIFIED MIGHT GET WIPED OUT!!!

# USAGE:
#       git archive HEAD | tar-checkout-f --yes-I-know /path/to/deploydir

die() { echo "$@" >&2; exit 1; }
# check arg-1 for CYA
dryrun=1
[ "$1" = "--yes-I-know" ] && { dryrun=0; shift; }
# check arg-1 is a valid directory
[ -d $1 ] || die "urkk (makes strangled noise)..."

td=$(mktemp -d)
# echo td=$td >&2
trap "/bin/rm -rf $td" 0

tee >(tar tf - | sed -e 's,^\./,,' > $td/tf) | tar -C "$1" -xf -

# ok now the fun stuff; delete stuff that should not be there
cd "$1"
cmd="echo would delete:"
[ "$dryrun" = "0" ] && cmd="rm -v"
find . -type f | sed -e 's,^\./,,' | fgrep -v -x -f $td/tf | xargs -r -d '\n' $cmd

Re: recipe to use git for deployment

[Neal Kreitzinger]

On 5/5/2012 7:14 AM, Sitaram Chamarty wrote:
> On Sat, May 5, 2012 at 11:00 AM, Neal Kreitzinger
> <nkreitzinger@gmail.com>  wrote:
>
>> Interestingly, git-archive was not intended for deployment, but it does
>> pretty good at it.
> Except it doesn't delete files that should be deleted (the way 'git
> checkout' does or rsync would if you gave it --delete).
>
Oh, yeah.  Thats why I move everything to a backup directory before 
untarring it.  That also satisfies our pre-git practice of saving off 
the old version before replacing it so we have a quick recovery point if 
the new version turns out to be worse than the old one!

v/r,
neal