Project- vs. Package-Level Branching in Git

Project- vs. Package-Level Branching in Git

[Thomas Hauk]

Back when I worked at a large games company, we used Perforce, and our repo structure looked a little something like this:

/branches
	/alpha
	/beta
/mainline
	/packages
		/external
			/foolib
				/1.0
				/1.1
				/2.0
		/internal
			/barlib
				/dev
				/1.0
				/2.0
			/bazlib
				/2.34
				/2.35
			/qux
				/dev

At the package level, we would split up packages/libraries into two groups based on if they were developed at the company or not (external/internal), and inside each one, we might have multiple versions. In the example above, the repo is for the "qux" game, which uses an internal "bazlib" library developed by another group, and the "barlib" library which was developed for use on "qux" and may be used simultaneously on other projects.

Project-level branches took mainline as a base and branched into the /branches directory. Package-level branches would usually take the "dev" version (which represented the current development version, akin to "master" in Git) as a base and branch into the same parent directory but with an actual version name.

I've successfully used this repo structure with several other projects over the years at other companies (who were mostly using Subversion). Now I'm trying to get into the Git swing of things, but it seems to be that Git only offers project-level branching, and doesn't allow for the kind of package-level branching I'm describing here.

Am I incorrect or is there a way to accomplish with Git what I was doing before with P4 and SVN?

--
Thomas Hauk
Shaggy Frog Software
www.shaggyfrog.com

Re: Project- vs. Package-Level Branching in Git

[Andreas Ericsson]

On 01/27/2011 08:38 PM, Thomas Hauk wrote:
> Back when I worked at a large games company, we used Perforce, and
> our repo structure looked a little something like this:
> 
> /branches /alpha /beta /mainline /packages /external /foolib /1.0 
> /1.1 /2.0 /internal /barlib /dev /1.0 /2.0 /bazlib /2.34 /2.35 /qux 
> /dev
> 
> At the package level, we would split up packages/libraries into two
> groups based on if they were developed at the company or not
> (external/internal), and inside each one, we might have multiple
> versions. In the example above, the repo is for the "qux" game, which
> uses an internal "bazlib" library developed by another group, and the
> "barlib" library which was developed for use on "qux" and may be used
> simultaneously on other projects.
> 
> Project-level branches took mainline as a base and branched into the
> /branches directory. Package-level branches would usually take the
> "dev" version (which represented the current development version,
> akin to "master" in Git) as a base and branch into the same parent
> directory but with an actual version name.
> 
> I've successfully used this repo structure with several other
> projects over the years at other companies (who were mostly using
> Subversion). Now I'm trying to get into the Git swing of things, but
> it seems to be that Git only offers project-level branching, and
> doesn't allow for the kind of package-level branching I'm describing
> here.
> 
> Am I incorrect or is there a way to accomplish with Git what I was
> doing before with P4 and SVN?
> 

You're correct. There's no way to accomplish what you wish to do
inside git. I'd suggest submodules, but that still doesn't get
you the per-directory branching inside a git repository, even
though it does make it rather trivial to change versions at will.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Project- vs. Package-Level Branching in Git

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 27, 2011 at 20:38, Thomas Hauk <tom@shaggyfrog.com> wrote:

> Am I incorrect or is there a way to accomplish with Git what I was
> doing before with P4 and SVN?

Different systems have different methodologies. Perforce and
Subversion are more of a virtual filesystem, while Git tries to track
a single project.

You'll be much better off if you have project-specific repositories.

Re: Project- vs. Package-Level Branching in Git

[Thomas Hauk]

On Jan 27, 2011, at 12:53 PM, Ævar Arnfjörð Bjarmason wrote:
> You'll be much better off if you have project-specific repositories.

But how often do you have a project that has no external or internal dependencies on any other packages or libraries? Any project I've ever done, big or small, has relied on some existing codebase. Imagine a project that uses liba and libb, which both reference libc. To use Git, I'd have to have copies of libc existing in three repositories, and copies of liba and lib in two repositories each. What a nightmare... and that's only a trivial hypothetical example.

I understand that Git was designed with a specific feature set in mind -- to manage Linux kernel development -- and as such isn't going to satisfy everyone. But I'm having trouble figuring out how to integrate it as the SCM in my projects, which aren't organized any differently than any other projects I've seen. Surely I can't be the only person with these difficulties? Git just doesn't seem to scale when it comes to componentized projects. Do other distributed VCSs work better than Git in this respect?

I'm really trying to get on the Git bandwagon, here.

--
Thomas Hauk
Shaggy Frog Software
www.shaggyfrog.com

Re: Project- vs. Package-Level Branching in Git

[Jay Soffian]

On Thu, Jan 27, 2011 at 6:22 PM, Thomas Hauk <tom@shaggyfrog.com> wrote:
> Git just doesn't seem to scale when it comes to componentized projects.

With git the canonical approach, such that it is, is for each
component to be its own repo. You then tie the repos together into a
project using submodules:

- http://progit.org/book/ch6-6.html
- http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html

However, submodules can be a challenge to work with, and so there have
been other attempts at tying multiple repos together such as:

- http://source.android.com/source/git-repo.html
- https://github.com/apenwarr/git-subtree

j.

Re: Project- vs. Package-Level Branching in Git

[Jakub Narebski]

Thomas Hauk <tom@shaggyfrog.com> writes:
> On Jan 27, 2011, at 12:53 PM, Ævar Arnfjörð Bjarmason wrote:

> > You'll be much better off if you have project-specific repositories.
> 
> But how often do you have a project that has no external or internal
> dependencies on any other packages or libraries? Any project I've
> ever done, big or small, has relied on some existing
> codebase. Imagine a project that uses liba and libb, which both
> reference libc. To use Git, I'd have to have copies of libc existing
> in three repositories, and copies of liba and lib in two
> repositories each. What a nightmare... and that's only a trivial
> hypothetical example.

That is only if lib{a,b,c} is _internal_ dependency.  In usual case
project A might depend on library B *to be installed*, but it doesn't
mean that source code of library B has to be in repository for project
A.  And in usual case when project A depends on library B, it relies
on library B public stable API (its build system might check if you
have new enough library B installed).  If you depend on specific
version of library, patched, that is your problem...

In the case of internal dependency, where you co-develop both project
A and library B, it makes most sense to have separate repositories for
A and for B, and tie them up using submodules or subtree merge.


In the case you describe you would need three *checkouts* of libc (if
checkout is needed for development, see above), but not necessary
three copies (three clones) of libc *repository* - one copy should be
enough.

> 
> I understand that Git was designed with a specific feature set in
> mind -- to manage Linux kernel development -- and as such isn't
> going to satisfy everyone. But I'm having trouble figuring out how
> to integrate it as the SCM in my projects, which aren't organized
> any differently than any other projects I've seen.

Well, you are braindamaged by your SCM ;-) ... just kidding.

Take a look how LibreOffice (Go-OOo), KDE, GNOME, GNU SourceMage Linux
distribution organize their repositories -- all of them are highly
modular / componentized.

> Surely I can't be the only person with these difficulties? Git just
> doesn't seem to scale when it comes to componentized projects. Do
> other distributed VCSs work better than Git in this respect?

Perhaps you could abuse Bazaar in that fashion, but I think most DVCS
use branching of whole project (branch in repository) model, rather
than the branching is copying (branch is folder) model that Perforce
and Subversion use.

Whole project branching has just too many advantages, or conversely
branch as folder (directory) has too many disadvantages.
 
> I'm really trying to get on the Git bandwagon, here.

World domination is not on agenda :-) The main goal of Git is to make
it easy for open-source projects to collaborate.

P.S. There are commercial DVCS: BitKeeper, Vault,...

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Project- vs. Package-Level Branching in Git

[Marc Branchaud]

On 11-01-27 06:22 PM, Thomas Hauk wrote:
> On Jan 27, 2011, at 12:53 PM, Ævar Arnfjörð Bjarmason wrote:
>> You'll be much better off if you have project-specific repositories.
> 
> But how often do you have a project that has no external or internal
> dependencies on any other packages or libraries? Any project I've ever
> done, big or small, has relied on some existing codebase. Imagine a
> project that uses liba and libb, which both reference libc. To use Git,
> I'd have to have copies of libc existing in three repositories, and copies
> of liba and lib in two repositories each. What a nightmare... and that's
> only a trivial hypothetical example.

Let me try to give you a more positive response...  :)

Where I work, our products rely on a vast amount of code from different
sources, not just various internal and external libraries but also the entire
FreeBSD tree as well as assorted forks of different Linux kernel versions
(because different customers have their own tweaked kernels that we need to
work with).

We use git in a variety of ways to manage all this.  We rely a lot on git's
submodule feature, but not exclusively.

For most external code, including the Linux kernel forks, we usually set up
an internal git mirror of whatever public repository the code has.  So each
external code base has its own git repository, and we attach those
repositories to our main repo using submodules.

This works fairly well, especially because we don't update the external
sources very often.  Working with submodules takes a bit of getting used to,
but it works nicely when the different pieces are reasonably independent, and
this is usually the case for external libraries.

For our internal libraries, we just keep them all in the main repository.
Our internal code does have some inter-dependencies, so it's convenient to
track them all together.  Note that this doesn't prevent internal libraries
from evolving independently -- even though a branch applies to the whole
repo, the branch's _topic_ can just be about one specific library.  Anyone
who needs to use the library's updated code can merge that branch into their
own, or base their work on that branch.  Eventually the library's branch gets
merged back into the mainline branch and everyone gets to use the updated code.

In addition to all that, we have a different way of working with the FreeBSD
code base.  This was put together a few years ago, and I would do it
differently now, but I'll describe it to give you an idea of what else is
possible with git.

The FreeBSD code tree lives in a subdirectory of our main repo.  It's not a
submodule or anything fancy, it's just the code.  We've modified the FreeBSD
code quite a bit, but we also keep it up to date with changes the FreeBSD
guys make.  We put all of their changes in a separate branch in our main
repo, and a script keeps that branch up to date.  When we're ready, we merge
the FreeBSD changes into our mainline branch.

Keeping that FreeBSD branch up to date is a bit involved.  The branch
actually reflects a single branch in the FreeBSD public subversion
repository.  We use "git svn" to maintain a git mirror of that subversion
branch, and what our script does is compare the latest FreeBSD subversion
revision number in our main repo to the one in the mirror ("git svn" records
the subversion revision numbers in the git commits it creates).  When the
script finds that the main repo is out of date, it extracts the patches for
each individual subversion commit and applies them as git commits in our main
repo's FreeBSD branch.  (Any git veterans who've read this far are probably
cringing right now...)

Anyway, my point is that git provides a lot of flexibility to let you work
with your code base in many different ways, but none of them are how
subversion or perforce work.  Coming from those tools, you have to shift your
mindset a bit to make the best use of git.  That can be frustrating, and I
won't say that git's model is The One True Way, but I've found that what feel
like limitations from the perspective of other tools usually turn out to be
relatively inconsequential.

		M.

Re: Project- vs. Package-Level Branching in Git

[in-gitvger]

In message <15B7CA2E-C584-4563-B9E3-D80861CD9565@shaggyfrog.com>, Thomas Hauk w
rites:

    On Jan 27, 2011, at 12:53 PM, Ævar Arnfjörð Bjarmason wrote:
    > You'll be much better off if you have project-specific repositories.

    But how often do you have a project that has no external or
    internal dependencies on any other packages or libraries? Any
    project I've ever done, big or small, has relied on some existing
    codebase. Imagine a project that uses liba and libb, which both
    reference libc. To use Git, I'd have to have copies of libc
    existing in three repositories, and copies of liba and lib in two
    repositories each. What a nightmare... and that's only a trivial
    hypothetical example.

Including libs in the superproject is the subtree merge method.  It
certainly works and provides inband commit exploration (since one repo
can see all commits), but it inconvenient to update and even harder to
export changes back to share with other liba users.  It may also cause
the repo to be inconveniently large.  Arranging for the correct
commits to be on the differently named branches (between the
subproject and the superproject) is also not convenient.

git-submodule is the normal approach for the problem you have.  There
is a strong binding from each commit in the superproject to commits in
the subprojects.  What is inconvenient is managing what branch you
need to check out on the subproject in order to get or make the right
changes in the right place.  It is also annoying if you are performing
active development on the subprojects since you continually have to
update the superproject and then recheckout the correct branches on
the subproject.

Another solution is gitslave (http://gitslave.sf.net).  This provides
a loose binding from the superproject to the subprojects which is very
convenient if you are doing active development on all of the
subprojects.  Specifically there is only a strong binding when you tag
(since you tag the superproject and all subprojects at the same time).
Generally, however, you check out the same branch/tag on all branches
at the same time, which obviously does not match your preferred usage,
except it gave me an idea.  Specifically, you could have your own
local master bare repositories for those packages and an orthogonal
naming scheme for branches and tags.  So the project foo would might
have branch foo-2.0 and liba libb and libc would all have those
branches as well.  When you want to update libb, a repo with the true
master upstream and the local master upstream would fetch the true
master and merge the changes from the correct branch into foo-2.0
and then push to the local upstream master.  Everyone else would then just
`gits pull` and would get the changes.

Of course this concept for a local master would work for submodules as
well, depending on whether you want the tight binding and
update/change annoyance or the loose binding and easier
update/changes.

					-Seth Robertson

Re: Project- vs. Package-Level Branching in Git

[Eugene Sajine]

> But how often do you have a project that has no external or internal dependencies on any other packages or libraries? Any project I've ever done, big or small, has relied on some existing codebase. Imagine a project that uses liba and libb, which both reference libc. To use Git, I'd have to have copies of libc existing in three repositories, and copies of liba and lib in two repositories each. What a nightmare... and that's only a trivial hypothetical example.
>
...> I'm really trying to get on the Git bandwagon, here.
>
> --
> Thomas Hauk
> Shaggy Frog Software
> www.shaggyfrog.com
>

For example at my shop we have very "component oriented" approach
(JAVA). Each project is a separate git repository, that is producing
one artifact (.jar, .war etc) and has a ivy dependency descriptor. We
are using Hudson CI to perform intergration builds of projects
themselves upon push and their downstream projects as well. In this
particular example there is no need to keep any copies that you're
talking about. For this Git works perfectly!!

OTOH there is a part of development that is using C++. And the whole
infrastructure is about static linking and is so heavily depending on
the CVS ability to expand keywords, that there is no way  (at least so
far i could not find an acceptable solution) to migrate the C++
development to Git.

My question is why do think you should have the copies??? Can it be
that the inability to use Git for your projects is related to the way
how you do things? May be you just have to be ready for the paradigm
shift first hand?

Thanks,
Eugene

Re: Project- vs. Package-Level Branching in Git

[Enrico Weigelt]

* Thomas Hauk <tom@shaggyfrog.com> wrote:
> Back when I worked at a large games company, we used Perforce, and
> our repo structure looked a little something like this:
> 
> /branches
> 	/alpha
> 	/beta
> /mainline
> 	/packages
> 		/external
> 			/foolib
> 				/1.0
> 				/1.1
> 				/2.0
> 		/internal
> 			/barlib
> 				/dev
> 				/1.0
> 				/2.0
> 			/bazlib
> 				/2.34
> 				/2.35
> 			/qux
> 				/dev

Ah, you're conditioned by a VCS that mixes up directory copies and branches,
just like SVN or TFS - those folks (VCS developers as well as their userbase)
still don't get the idea that these are orthogonal concepts ;-o
Needless to say that this makes working w/ branches quite hard (compared
to git ;-p), not even to mention things like remote tracking, rebase, etc.
(guess what: M$ itself recommends against branches or at least very beaurocratic
planning and processes in their own official TFS papers ;-p)

Just forget about that all, forget the whole idea of having dozens of
packages in one fat tree. Develop and track each package saparately and
use an separate dist build tool (eg. Briegel) to put things together.

It'll take some time to get your head around this ideology, but it'll pay out
if you do it consequently. (I've already been through this some years ago).

> At the package level, we would split up packages/libraries into two groups
> based on if they were developed at the company or not (external/internal),

Why exactly that split ? Why not having each package separately, independent
on the supplier ? Each package should stand completely on its own (assuming
dependencies resolved), or it isn't really a package but just a bunch of files.

> and inside each one, we might have multiple versions. In the example above,
> the repo is for the "qux" game, which uses an internal "bazlib" library
> developed by another group, and the "barlib" library which was developed
> for use on "qux" and may be used simultaneously on other projects.

That qualifies an _separate_ (generic) package, which should be developed
and deployed separately. If you need to tweak it for (potentially) each
project (so, have per-project branches of it), you'll most likely have
some serious architectural flaw.

> I've successfully used this repo structure with several other projects
> over the years at other companies (who were mostly using Subversion).

My current customer also used such a model (previously in PVCS, not TFS),
and they discuss hours over hours about how and when to branch and whether
certain things could be merged back at all, instead of just doing the
actual work.

For the product I'm currently working on (imagine that: an embedded linux
project that gets tracked via TFS ;-o) they always tried to do the projects
(each on essentially the same codebase, but each job/project has some
customer-specific changes) sequentially (and even mixed up with standard
feature development), so they could lie to themselves that they would
never need branches. In the end they had two completely separate 
incompatible codebasis which now have to be "merged" together manually,
consuming several man-month. (now I'm there to clean up the mess ;-o).

> Now I'm trying to get into the Git swing of things, but it seems to be
> that Git only offers project-level branching, and doesn't allow for the
> kind of package-level branching I'm describing here.

On sourcetree-, not project-level. Projects and sourcetrees are
completely different concepts. Another important point that many
folks (especially IDE designers) never really understood.

> Am I incorrect or is there a way to accomplish with Git what I was
> doing before with P4 and SVN?

Not really. Rethink your project organization and workflows.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: Project- vs. Package-Level Branching in Git

[Enrico Weigelt]

* Thomas Hauk <tom@shaggyfrog.com> wrote:

> Imagine a project that uses liba and libb, which both reference libc.
> To use Git, I'd have to have copies of libc existing in three
> repositories, and copies of liba and lib in two repositories each. 

No, just leave them as they are: separate packages. Period.
Let a dist buildsystem handle the project's build process,
including dependency resolving, packaging, etc.

> Git just doesn't seem to scale when it comes to componentized
> projects. 

These "componentized" projects (as you call it - I'd rather call
it the exact opposite) suffer from an ideology that doens't know
to differenciate between the fundamental stages of an software
building up to deployment, they try do to everything in one fat
blob - that's why fail or burn expensive resources frequently.
(I've seen that often enough in my last 15 years of software
engineering ...)


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: Project- vs. Package-Level Branching in Git

[Enrico Weigelt]

* Jakub Narebski <jnareb@gmail.com> wrote:

> That is only if lib{a,b,c} is _internal_ dependency.  In usual case
> project A might depend on library B *to be installed*, but it doesn't
> mean that source code of library B has to be in repository for project
> A.  And in usual case when project A depends on library B, it relies
> on library B public stable API (its build system might check if you
> have new enough library B installed).  If you depend on specific
> version of library, patched, that is your problem...

To make it more clear: 

At buildtime, a _package_ (not project!) "A" requires a already built
and installed package B in some sane place where the toolchain can find it.
On deployment of package "A", it has to be made sure that package "B"
is also deployed (eg. by a dependency-handling package manager).

These are two entirely separate stages in a software's lifecycle.
Buildtime and deployment dependencies may be different (even deployment
and runtime deps may differ).

> In the case of internal dependency, where you co-develop both project
> A and library B, it makes most sense to have separate repositories for
> A and for B, and tie them up using submodules or subtree merge.

I, personally, wouldn't use submodules - too complicated. Instead have
just one tree per package(-variant) and keep these completely separate.

> > I understand that Git was designed with a specific feature set in
> > mind -- to manage Linux kernel development -- and as such isn't
> > going to satisfy everyone. But I'm having trouble figuring out how
> > to integrate it as the SCM in my projects, which aren't organized
> > any differently than any other projects I've seen.
> 
> Well, you are braindamaged by your SCM ;-) ... just kidding.
> 
> Take a look how LibreOffice (Go-OOo), KDE, GNOME, GNU SourceMage Linux
> distribution organize their repositories -- all of them are highly
> modular / componentized.

Well, I wouldn't say LO is really modularized, yet. (but we're
working on that ...).
 

cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: Project- vs. Package-Level Branching in Git

[David Aguilar]

On Sat, Jan 29, 2011 at 08:42:59PM +0100, Enrico Weigelt wrote:
> * Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > That is only if lib{a,b,c} is _internal_ dependency.  In usual case
> > project A might depend on library B *to be installed*, but it doesn't
> > mean that source code of library B has to be in repository for project
> > A.  And in usual case when project A depends on library B, it relies
> > on library B public stable API (its build system might check if you
> > have new enough library B installed).  If you depend on specific
> > version of library, patched, that is your problem...
> 
> To make it more clear: 
> 
> At buildtime, a _package_ (not project!) "A" requires a already built
> and installed package B in some sane place where the toolchain can find it.
> On deployment of package "A", it has to be made sure that package "B"
> is also deployed (eg. by a dependency-handling package manager).
> 
> These are two entirely separate stages in a software's lifecycle.
> Buildtime and deployment dependencies may be different (even deployment
> and runtime deps may differ).

This is exactly how we do it at my workplace.  We have literally
hundreds of individual git repositories.  Naturally, some
packages depend on others and the only "trick" is building them
in the correct dependency order.  A simple dependency tree
handles this for us.

We use same-named branches across several repos when coordinating
features across many projects.  e.g. we had an "el6" branch
when we were gettings things ready for that platform.  It's a
convention but it helps when writing helper scripts.

We can clone and work on any subset of the entire tree by
cloning just the repos we are interested in.  Setting
$LD_LIBRARY_PATH and $PATH helps when testing builds in their
sandboxes.  You still need to get the compiler/linker to
construct paths into the sandboxes instead of the standard
release area.

This is what the pkg-config command does.  It respects the
$PKG_CONFIG_PATH environment variable which can be used to
point to staged installs so that you don't have to deploy the
package before building against it.  The idea is so simple
that you could write an equivalent command in an afternoon and
extend it to work however you need in the event that pkg-config
does not fit.

There's only two components that are needed to make this work:

1. a simple shell wrapper to aid in setting the env. variables
   on the fly.  Let's call it "devo".  That way you can say
   "devo foo bar baz" and it'll put
   foo/linux-x64/lib64/pkgconfig and bar/linux-x64/lib64/pkgconfig
   (assuming linux-x64 is the "make install"ed path)
   at the front of the PKG_CONFIG_PATH, foo/linux-x86/bin in PATH,
   etc.

   You can do without a wrapper by either setting the variables
   manually or by having a shell scriplet that you 'source'
   whenever you want the settings.

2. the build must use the pkg-config command when constructing
   include/library paths.

As I mentioned, perhaps you don't want to use pkg-config.
The idea ("overridable projects using a single env. variable")
is the important part.  You can certainly write something
tailored to your needs if you want something simpler.
-- 
		David

Re: Project- vs. Package-Level Branching in Git

[Enrico Weigelt]

* David Aguilar <davvid@gmail.com> wrote:

Hi,

> This is exactly how we do it at my workplace.  We have literally
> hundreds of individual git repositories.  Naturally, some
> packages depend on others and the only "trick" is building them
> in the correct dependency order.  A simple dependency tree
> handles this for us.

perhaps you'd like to have a look at my Briegel build tool:

    git://pubgit.metux.de/projects/briegel.git
    
;-)

> We use same-named branches across several repos when coordinating
> features across many projects.  e.g. we had an "el6" branch
> when we were gettings things ready for that platform.  It's a
> convention but it helps when writing helper scripts.

Did you have these branches for all packages ?

> We can clone and work on any subset of the entire tree by
> cloning just the repos we are interested in.  Setting
> $LD_LIBRARY_PATH and $PATH helps when testing builds in their
> sandboxes.  You still need to get the compiler/linker to
> construct paths into the sandboxes instead of the standard
> release area.

I'd suggest pushing everything through a sysroot'ed crosscompiler
and only install the absolute required dependencies in the sysroot
on each package build. This tends to show up a lot of programming
errors that otherwise stay unnoticed for a long time.
(Briegel goes exactly that way and handles it automatically ;-p)
 
> This is what the pkg-config command does.  It respects the
> $PKG_CONFIG_PATH environment variable which can be used to
> point to staged installs so that you don't have to deploy the
> package before building against it. 

With sysroot, it's even a bit more cleaner, pkg-config can handle
the path fixups automatically then.

> The idea is so simple that you could write an equivalent command
> in an afternoon and extend it to work however you need in the
> event that pkg-config does not fit.

Actually, I only know of rare cases where pkg-config doesn't
really fit. Mostly due bad software design. Once thing I'm yet
missing is something pkg-config alike which replaces most of
the autofool-tests (eg. whether the target supports some
syscall, stack directions, etc).

> 2. the build must use the pkg-config command when constructing
>    include/library paths.

Still there're lots of packages which dont use pkg-config.
Some of those I'm already fixing in my OSS-QM project.
(Everybody's invited to join in ;-))


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: Project- vs. Package-Level Branching in Git

[Jakub Narebski]

Enrico Weigelt <weigelt@metux.de> writes:
> * David Aguilar <davvid@gmail.com> wrote:
> 
> > This is exactly how we do it at my workplace.  We have literally
> > hundreds of individual git repositories.  Naturally, some
> > packages depend on others and the only "trick" is building them
> > in the correct dependency order.  A simple dependency tree
> > handles this for us.
> 
> perhaps you'd like to have a look at my Briegel build tool:
> 
>     git://pubgit.metux.de/projects/briegel.git
>     
> ;-)

Is it Git-specific build tool?  If it is so, perhaps it would be good
to add it to
 
  https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools

page on Git Wiki?

Thanks in advance
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Project- vs. Package-Level Branching in Git

[Enrico Weigelt]

* Jakub Narebski <jnareb@gmail.com> wrote:

> > perhaps you'd like to have a look at my Briegel build tool:
> > 
> >     git://pubgit.metux.de/projects/briegel.git
> >     
> > ;-)
> 
> Is it Git-specific build tool?  If it is so, perhaps it would be good
> to add it to
>  
>   https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools
> 
> page on Git Wiki?

Not really, it just happens to use git extensively, eg. it can
pull sources and sysroot images directly from git repos (Tarballs
and textbased patches are still supported, but I dont use them
anymore for years).


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------