Linking multiple Git repositories for version tracking

Linking multiple Git repositories for version tracking

[James Beck]

Hi,

I'm developing firmware that is composed of multiple projects. Each  
section of the firmware has it's own git repository (each section  
correlates to a physical processor). But the firmware as a whole is  
getting to the point where I have to remember which version of Firmware A  
is compatible with Firmware B. If I add a feature to B that requires some  
tweaks in A, I need to know that both A v3.04 and B v2.7 need to be used  
together.

I'm starting to move into alpha with this code, so I really need to have a  
system that keeps track of compatible firmware versions. The best I can  
come up with is a plain text file (or Excel spreadsheet) that lists the  
overall firmware version, and the Git hash for each individual project.  
That way, if I want to load up a particular firmware version, I can  
checkout each hash for that version. Seems foolproof, but not terribly  
easy, and somewhat annoying.

I know submodules might be used, but it's not super obvious how to make  
their paradigm work nicely here. (You check out a version you want, and  
then list all the submodule git hashes for that version? What happens if  
one hash needs updating? Do you branch it?) It seems more complicated than  
I'd like.

But this can't be a problem that only I have. What have you done to keep  
track of software bundle versions?

Thanks!
James

Re: Linking multiple Git repositories for version tracking

[Avery Pennarun]

On Fri, Jan 8, 2010 at 12:03 PM, James Beck <james@jmbeck.com> wrote:
> I'm developing firmware that is composed of multiple projects. Each
> section of the firmware has it's own git repository (each section
> correlates to a physical processor). But the firmware as a whole is
> getting to the point where I have to remember which version of Firmware A
> is compatible with Firmware B. If I add a feature to B that requires some
> tweaks in A, I need to know that both A v3.04 and B v2.7 need to be used
> together.
>
> I'm starting to move into alpha with this code, so I really need to have a
> system that keeps track of compatible firmware versions. The best I can
> come up with is a plain text file (or Excel spreadsheet) that lists the
> overall firmware version, and the Git hash for each individual project.
> That way, if I want to load up a particular firmware version, I can
> checkout each hash for that version. Seems foolproof, but not terribly
> easy, and somewhat annoying.
>
> I know submodules might be used, but it's not super obvious how to make
> their paradigm work nicely here. (You check out a version you want, and
> then list all the submodule git hashes for that version? What happens if
> one hash needs updating? Do you branch it?) It seems more complicated than
> I'd like.

This seems like exactly what submodules were designed to do.

1. create a "superproject" for each physical product

2. use submodules to link to the right firmware versions for that product

3. when you make a new release of that physical product, update the
firmware links.

4. when someone wants to check out a particular version of the
product, they retrieve the product's repo and ask git submodule to
checkout the submodules.

Which part is not working for you?

Have fun,

Avery

Re: Linking multiple Git repositories for version tracking

[Michael Witten]

On Fri, Jan 8, 2010 at 11:03 AM, James Beck <james@jmbeck.com> wrote:
> I'm starting to move into alpha with this code, so I really need to have a
> system that keeps track of compatible firmware versions. The best I can
> come up with is a plain text file (or Excel spreadsheet) that lists the
> overall firmware version, and the Git hash for each individual project.
> That way, if I want to load up a particular firmware version, I can
> checkout each hash for that version. Seems foolproof, but not terribly
> easy, and somewhat annoying.

The Xorg project is currently composed of 197 separate `modules', and
I believe that (practically) each has its own Git repo.

Their solution seems to be the exact same as yours, though it involves
translating between official release numbers and commits from what I
recall, all of which uses this hand-written list:

    http://cgit.freedesktop.org/xorg/util/modular/tree/module-list.txt

So, perhaps you just need to write a small script to do the checkouts
automatically for you.

Also, you could use Git tags rather than raw Git hashes. If you have
"releases", you could just make the same tag name in each Git repo, so
that checking out any particular "release" just amounts to having a
script loop through the repos and checkout the necesarry commit using
the associated tag. In fact, I suggested this to the Xorg guys not
long ago:

    http://www.mail-archive.com/xorg-devel@lists.x.org/msg03037.html

See:

    http://cgit.freedesktop.org/xorg/util/modular/tree/
    http://www.x.org/wiki/Development/Documentation/ReleaseHOWTO

Re: Linking multiple Git repositories for version tracking

[Christoph Jahn]

I have had the very same issue a while ago when still using SVN (shame on 
me ;-). SVN offers a special tag "svn:externals" which is something like 
symlinks. Having used it for a while I came to the conclusion that such a 
thing, as well-intentioned as it may be, is something I want to avoid by 
all means.

There are two reasons for that:
- Lock-in
- It is conceptually wrong

The conceptual part -which I consider to be more important- is mostly 
about the fact that we are talking about dependency management and not 
version control any more. Do I really want to "hide" dependencies inside 
the VCS? Also, there may be different dependencies for compilation and 
testing. That's why I chose to use Maven for this task.

My $0.02

-- 
Best regards,
Christoph