Looking for a way to build latest tagged releases in recipes

Looking for a way to build latest tagged releases in recipes

[Pascal Huerst]

Hey everyone!

I'm looking for a way in yocto to build tagged releases or actually to
build the newest tagged release from repositories.

There is a way to specify a branch or a tag in a recipe, but no way to
always clone the newest tagged release, that I'm aware of. I submitted a
patch to bitbake's mailing list with a proposal [1], that would use
bitbake's `latest_versionstring()`, to always find, fetch and build the
latest tagged release. Unfortunately there wasn't much of a reaction on
that proposal, so I'm taking a step back, collecting ideas on how this
could be achieved properly.

Please share your ideas...

Thanks!
Pascal

[1]
http://lists.openembedded.org/pipermail/bitbake-devel/2019-December/020618.html

Re: Looking for a way to build latest tagged releases in recipes

[Khem Raj]

On Wed, Jan 8, 2020 at 4:10 AM Pascal Huerst <pascal.huerst@gmail.com> wrote:
>
> Hey everyone!
>
> I'm looking for a way in yocto to build tagged releases or actually to
> build the newest tagged release from repositories.
>
> There is a way to specify a branch or a tag in a recipe, but no way to
> always clone the newest tagged release, that I'm aware of. I submitted a
> patch to bitbake's mailing list with a proposal [1], that would use
> bitbake's `latest_versionstring()`, to always find, fetch and build the
> latest tagged release. Unfortunately there wasn't much of a reaction on
> that proposal, so I'm taking a step back, collecting ideas on how this
> could be achieved properly.
>
> Please share your ideas...
>

while I understand the usecase, it has a downside, where a build can fail due to
someone pushing a new tag. I think we allow AUTOREV on branches which
is helpful in development and
CI systems. but deducing newest tag is useful when making releases,
this also means that it will bump the rev
in recipe automatically and people might get enticed to use it in
release process.

For such usecases, Maybe you can use floating tags ( like
'latest-release' or 'last-good' ) and move them when needed and lock
the recipes to use them to achieve what you want.

since use of AUTOREV in public recipes is discouraged, this does not
affect public layers as much
but I do worry about alternate workflows.

> Thanks!
> Pascal
>
> [1]
> http://lists.openembedded.org/pipermail/bitbake-devel/2019-December/020618.html
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Looking for a way to build latest tagged releases in recipes

[Richard Purdie]

On Wed, 2020-01-08 at 13:10 +0100, Pascal Huerst wrote:
> I'm looking for a way in yocto to build tagged releases or actually
> to build the newest tagged release from repositories.
> 
> There is a way to specify a branch or a tag in a recipe, but no way
> to always clone the newest tagged release, that I'm aware of. I
> submitted a patch to bitbake's mailing list with a proposal [1], that
> would use bitbake's `latest_versionstring()`, to always find, fetch
> and build the latest tagged release. Unfortunately there wasn't much
> of a reaction on that proposal, so I'm taking a step back, collecting
> ideas on how this could be achieved properly.
> 
> Please share your ideas...

I did look at and think a bit about your patch. The trouble for me (as
the bitbake maintainer) is that:

a) it isn't clear/obvious from the variable name what it does (or even 
   from the code)
b) it adds yet more options to the fetcher which increases the test 
   matrix and possible ways things can break
c) the test doesn't actually test it does what its supposed to (no 
   check is made on which output is downloaded for a given revision)

Some of these can be fixed, some are much harder and I can't decide if
supporting this kind of edge case is going to be an overall good thing
for the project or not. I don't want to give you false hope by fixing
c) only to say "no".

Add in Khem's valid concerns about reproducibility and it makes it a
tough one even to give feedback on.

If we have a ton of users saying "yes we really need this", that would
help but I haven't seen that as yet...

Cheers,

Richard

Re: Looking for a way to build latest tagged releases in recipes

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2400 bytes --]

I think you can do this with a little help from devtool:

devtool upgrade component
devtool finish component layer-path
git add/commit layer-path
bitbake component

This has the added benefit of keeping things deterministic and reproducible.

You can further automate these steps with help from AUH:
https://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/about/

Alex

On Thu, 9 Jan 2020 at 11:52, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2020-01-08 at 13:10 +0100, Pascal Huerst wrote:
> > I'm looking for a way in yocto to build tagged releases or actually
> > to build the newest tagged release from repositories.
> >
> > There is a way to specify a branch or a tag in a recipe, but no way
> > to always clone the newest tagged release, that I'm aware of. I
> > submitted a patch to bitbake's mailing list with a proposal [1], that
> > would use bitbake's `latest_versionstring()`, to always find, fetch
> > and build the latest tagged release. Unfortunately there wasn't much
> > of a reaction on that proposal, so I'm taking a step back, collecting
> > ideas on how this could be achieved properly.
> >
> > Please share your ideas...
>
> I did look at and think a bit about your patch. The trouble for me (as
> the bitbake maintainer) is that:
>
> a) it isn't clear/obvious from the variable name what it does (or even
>    from the code)
> b) it adds yet more options to the fetcher which increases the test
>    matrix and possible ways things can break
> c) the test doesn't actually test it does what its supposed to (no
>    check is made on which output is downloaded for a given revision)
>
> Some of these can be fixed, some are much harder and I can't decide if
> supporting this kind of edge case is going to be an overall good thing
> for the project or not. I don't want to give you false hope by fixing
> c) only to say "no".
>
> Add in Khem's valid concerns about reproducibility and it makes it a
> tough one even to give feedback on.
>
> If we have a ton of users saying "yes we really need this", that would
> help but I haven't seen that as yet...
>
> Cheers,
>
> Richard
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3313 bytes --]

Re: Looking for a way to build latest tagged releases in recipes

[Pascal Huerst]

On 09/01/2020 11:51, Richard Purdie wrote> I did look at and think a bit
about your patch. The trouble for me (as
> the bitbake maintainer) is that:
> 
> a) it isn't clear/obvious from the variable name what it does (or even 
>    from the code)
> b) it adds yet more options to the fetcher which increases the test 
>    matrix and possible ways things can break
> c) the test doesn't actually test it does what its supposed to (no 
>    check is made on which output is downloaded for a given revision)

I certainly wouldn't insist on the name and considered it a proposal,
but you're right, it's not obvious at all. As for test-ability, I can
relate to your concerns and understand that you don't want to maintain a
corner-case like this - Which leads me to the conclusion that we should
probably rethink our internal concept for release builds...

> Some of these can be fixed, some are much harder and I can't decide if
> supporting this kind of edge case is going to be an overall good thing
> for the project or not. I don't want to give you false hope by fixing
> c) only to say "no".

Understandable.

> Add in Khem's valid concerns about reproducibility and it makes it a
> tough one even to give feedback on.
> 
> If we have a ton of users saying "yes we really need this", that would
> help but I haven't seen that as yet...

Fair enough.

Thanks for your thoughts!

Re: Looking for a way to build latest tagged releases in recipes

[Richard Purdie]

On Thu, 2020-01-09 at 12:59 +0100, Pascal Huerst wrote:
> On 09/01/2020 11:51, Richard Purdie wrote> I did look at and think a bit
> about your patch. The trouble for me (as
> > the bitbake maintainer) is that:
> > 
> > a) it isn't clear/obvious from the variable name what it does (or even 
> >    from the code)
> > b) it adds yet more options to the fetcher which increases the test 
> >    matrix and possible ways things can break
> > c) the test doesn't actually test it does what its supposed to (no 
> >    check is made on which output is downloaded for a given revision)
> 
> I certainly wouldn't insist on the name and considered it a proposal,
> but you're right, it's not obvious at all. As for test-ability, I can
> relate to your concerns and understand that you don't want to maintain a
> corner-case like this - Which leads me to the conclusion that we should
> probably rethink our internal concept for release builds...
> 
> > Some of these can be fixed, some are much harder and I can't decide if
> > supporting this kind of edge case is going to be an overall good thing
> > for the project or not. I don't want to give you false hope by fixing
> > c) only to say "no".
> 
> Understandable.
> 
> > Add in Khem's valid concerns about reproducibility and it makes it a
> > tough one even to give feedback on.
> > 
> > If we have a ton of users saying "yes we really need this", that would
> > help but I haven't seen that as yet...
> 
> Fair enough.
> 
> Thanks for your thoughts!

FWIW I do want to see the system used to do things like this. I've kind
of envisaged you'd do it "up front" though. For example it should be
trivial for a script to use tinfoil, iterate through and generate an
include file of the revisions you want, using the fetcher calls.

Yes, its slightly more annoying to generate the config, then build it
but if it stops the core becoming unmaintainable with corner cases,
that could be the right decision.

If you do something like that it would be great to share it so others
can see what you did and be able to build off it...

Cheers,

Richard

Re: Looking for a way to build latest tagged releases in recipes

[Pascal Huerst]

On 09/01/2020 05:05, Khem Raj wrote:
> For such usecases, Maybe you can use floating tags ( like
> 'latest-release' or 'last-good' ) and move them when needed and lock
> the recipes to use them to achieve what you want.

I guess we will have to do it that way then, or we create some scripts
which create an include file of the revisions we want, using the fetcher
calls, as proposed by Richard.

> since use of AUTOREV in public recipes is discouraged, this does not
> affect public layers as much
> but I do worry about alternate workflows.
> 
>> Thanks!
>> Pascal
>>
>> [1]
>> http://lists.openembedded.org/pipermail/bitbake-devel/2019-December/020618.html
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Looking for a way to build latest tagged releases in recipes

[Pascal Huerst]

On 09/01/2020 13:08, Richard Purdie wrote:
> On Thu, 2020-01-09 at 12:59 +0100, Pascal Huerst wrote:
>> On 09/01/2020 11:51, Richard Purdie wrote> I did look at and think a bit
>> about your patch. The trouble for me (as
>>> the bitbake maintainer) is that:
>>>
>>> a) it isn't clear/obvious from the variable name what it does (or even 
>>>    from the code)
>>> b) it adds yet more options to the fetcher which increases the test 
>>>    matrix and possible ways things can break
>>> c) the test doesn't actually test it does what its supposed to (no 
>>>    check is made on which output is downloaded for a given revision)
>>
>> I certainly wouldn't insist on the name and considered it a proposal,
>> but you're right, it's not obvious at all. As for test-ability, I can
>> relate to your concerns and understand that you don't want to maintain a
>> corner-case like this - Which leads me to the conclusion that we should
>> probably rethink our internal concept for release builds...
>>
>>> Some of these can be fixed, some are much harder and I can't decide if
>>> supporting this kind of edge case is going to be an overall good thing
>>> for the project or not. I don't want to give you false hope by fixing
>>> c) only to say "no".
>>
>> Understandable.
>>
>>> Add in Khem's valid concerns about reproducibility and it makes it a
>>> tough one even to give feedback on.
>>>
>>> If we have a ton of users saying "yes we really need this", that would
>>> help but I haven't seen that as yet...
>>
>> Fair enough.
>>
>> Thanks for your thoughts!
> 
> FWIW I do want to see the system used to do things like this. I've kind
> of envisaged you'd do it "up front" though. For example it should be
> trivial for a script to use tinfoil, iterate through and generate an
> include file of the revisions you want, using the fetcher calls.

Sure. Writing some scripts to do the job was my first intend, but then I
was aiming for something that could potentially be merged upstream. But
I guess I underestimated the overall implications...

> Yes, its slightly more annoying to generate the config, then build it
> but if it stops the core becoming unmaintainable with corner cases,
> that could be the right decision.
> 
> If you do something like that it would be great to share it so others
> can see what you did and be able to build off it...

Sure. Will do.

Re: Looking for a way to build latest tagged releases in recipes

[chris.laplante]

Hi Richard,

> FWIW I do want to see the system used to do things like this. I've kind
> of envisaged you'd do it "up front" though. For example it should be
> trivial for a script to use tinfoil, iterate through and generate an
> include file of the revisions you want, using the fetcher calls.
> 
> Yes, its slightly more annoying to generate the config, then build it
> but if it stops the core becoming unmaintainable with corner cases,
> that could be the right decision.
> 
> If you do something like that it would be great to share it so others
> can see what you did and be able to build off it...

This is actually something I've been working on and thinking about a lot for the past year and a half. Our application layer has probably 30 or so recipes that use AUTOREV. For the purposes of reproducible builds, we need to generate similar output to what buildhistory-collect-srcrevs generates, e.g. 

	SRCREV_pn-my_daemon = "githash" 

I've gone back and forth and re-implemented the code to do so a half dozen times. Chronologically, these are the fundamental strategies I've attempted:

1. Simply in terms of buildhistory-collect-srcrevs. 
	Pros: dead simple
	Cons: requires that you actually build stuff first; requires buildhistory to be enabled
2. As a bbclass, injecting a task between do_fetch and do_unpack ("do_capture_srcrevs")
	Pros: can be invoked without actually building stuff, e.g. bitbake my-image --runall=do_capture_srcrevs
	Cons: adds yet another task to the recipe; a little ugly since you either need to INHERIT the class, or individually inherit it in recipes 
3. As a separate tinfoil script (as you suggest) which I run before the actual bitbake. 
	Pros: decoupled from the actual build
	Cons: since it's decoupled, it's slower - each recipe needs to be parsed
4. Hacking the srcrev cache (inside fetch2) to log all information necessary to generate the SRCREV overrides, paired with a script that reads the cache and does it
	Pros: Equally suitable for the "config-then-build" workflow ("bitbake --parse-only && generate_overrides.py >> local.conf && bitbake image") or the "build-then-config" workflow ("bitbake image && generate_overrides.py > overrides.conf")
	Cons: Invasive, required large changes to fetch2/__init__.py and some smaller changes to fetch2/git.py

In the current incarnation (#4), I also go a step further and, in cases where a SRC_URI has a parameterized branch, I generate BRANCH override lines as well. For example, if I see:

	MY_BRANCH ??= "master"
	SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1; branch=${MY_BRANCH}" 

I generate this line along with the corresponding SRCREV override:

	MY_BRANCH_pn-recipe = "master"

The algorithm for detecting the branch variable in the SRC_URI entry is as a little complicated, and not super relevant at this moment, but I want to get it off my chest :). The code first tries to find a parameterized branch by using a regex to find stuff like "branch=${VAR}". If that fails, it could be because the branch is expressed as something more complicated like this:

	BRANCH_RELEASE_NUMBER ??= "1.0"
	SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1; branch=release-${BRANCH_RELEASE_NUMBER}"

In that case, I fall back on a breadth-first search on all the variables that SRC_URI references (recursively) to find the "deepest" one that controls the branch. I wrote this in terms of d.expandWithRefs, and can detail it further if people are interested. Basically, I didn't want to enforce a particular convention on the name of the branch variable. You'd think BRANCH would cover it, but the kernel recipes also have KMETA and stuff like that. 

I realize the branch stuff as currently implemented is maybe too opinionated (and a bit dangerous) for inclusion in bitbake. But I digress. 

I'm very interested in discussing the SRCREV (and perhaps BRANCH) stuff further with anyone interested. I would love to see some strategy for doing this make its way into bitbake or poky, so I can stop reimplementing it and second guessing myself :).

Thanks,
Chris

Re: Looking for a way to build latest tagged releases in recipes

[chris.laplante]

> I've gone back and forth and re-implemented the code to do so a half dozen times. Chronologically, these are the fundamental
> strategies I've attempted:
> 
> 1. Simply in terms of buildhistory-collect-srcrevs.
> 	Pros: dead simple
> 	Cons: requires that you actually build stuff first; requires buildhistory to be enabled
> 2. As a bbclass, injecting a task between do_fetch and do_unpack ("do_capture_srcrevs")
> 	Pros: can be invoked without actually building stuff, e.g. bitbake my-image --runall=do_capture_srcrevs
> 	Cons: adds yet another task to the recipe; a little ugly since you either need to INHERIT the class, or individually inherit it in
> recipes
> 3. As a separate tinfoil script (as you suggest) which I run before the actual bitbake.
> 	Pros: decoupled from the actual build
> 	Cons: since it's decoupled, it's slower - each recipe needs to be parsed
> 4. Hacking the srcrev cache (inside fetch2) to log all information necessary to generate the SRCREV overrides, paired with a script that
> reads the cache and does it
> 	Pros: Equally suitable for the "config-then-build" workflow ("bitbake --parse-only && generate_overrides.py >> local.conf &&
> bitbake image") or the "build-then-config" workflow ("bitbake image && generate_overrides.py > overrides.conf")
> 	Cons: Invasive, required large changes to fetch2/__init__.py and some smaller changes to fetch2/git.py
> 
> In the current incarnation (#4), I also go a step further and, in cases where a SRC_URI has a parameterized branch, I generate BRANCH
> override lines as well. For example, if I see:

As mentioned in IRC, as I get deeper into it, I'll be going with approach #3 after all :/. Hope to post a tinfoil script soon. 

Chris

Re: Looking for a way to build latest tagged releases in recipes

[chris.laplante]

> > As mentioned in IRC, as I get deeper into it, I'll be going with approach #3 after all :/. Hope to post a tinfoil script soon.
> 
> That would be very much appreciated!

Here's a prototype bbclass: https://gist.github.com/mostthingsweb/39e24b329de70c0212e5bf21a431e143. Please give it a try.

All you need to do is INHERIT it in local.conf:
	INHERIT += "revrecord"

Then you can just do "bitbake --parse-only" and during parsing it will create a directory structure in ${TOPDIR}/revs that looks something like this:

revs
├── 20200116170750
│   ├── recipe1
│   │   └── revs.inc
│   ├── recipe2
│   │   └── revs.inc
│   ├── recipe3
│   │   └── revs.inc
│   └── revs.inc
└── latest -> 20200116170750

It also supports multiconfig. When multiconfig is enabled, there will be an additional layer of directories in the 'revs' directory, one per multiconfig (plus "default"):

revs
├── 20200116170750
│   ├── default
│   │   ├── recipe1
│   │   │   └── revs.inc
│   │   ├── recipe2
│   │   │   └── revs.inc
│   │   └── revs.inc
│   └── mc1
│       ├── recipe1
│       │   └── revs.inc
│       ├── recipe2
│       │   └── revs.inc
│       └── revs.inc
└── latest -> 20200116170750


TODO:

* Right now, the class triggers a base environment change every time, which means BitBake always reparses every recipe. I guess this is because I'm modifying the datastore when I get bb.event.ConfigParsed and bb.event.MultiConfigParsed, in order to ensure REVRECORD_DATETIME is the same across all configurations when multiconfig is active. Perhaps there is a more elegant way to do this that won't trigger the environment change? To be fair I think in most cases you won't care, since I expect this class to mainly be used in a continuous integration environment where you'll be doing a clean build and having to reparse everything anyway. But I could also see this class a useful to thing to always have enabled, even for personal builds, and in that case obviously I'd want to fix this issue.
* When multiconfig is active, I would also like a "global" revs.inc to be generated, which is the product of aggregated the "revs.inc" for all the multiconfigs. Still need to think about how exactly this will work in the face of conflicting SRCREVs. 
* We have a use case for JSON format data as well ("revs.json") - I'll add that too.
* Possibly a small tinfoil script that simply automates the task of INHERIT'ing this class, parsing all the recipes, and then dumping revs.inc.


Thanks,
Chris 

Re: Looking for a way to build latest tagged releases in recipes

[chris.laplante]

> TODO:
> 
> * Right now, the class triggers a base environment change every time, which means BitBake always reparses every recipe. I guess this
> is because I'm modifying the datastore when I get bb.event.ConfigParsed and bb.event.MultiConfigParsed, in order to ensure
> REVRECORD_DATETIME is the same across all configurations when multiconfig is active. Perhaps there is a more elegant way to do
> this that won't trigger the environment change? To be fair I think in most cases you won't care, since I expect this class to mainly be
> used in a continuous integration environment where you'll be doing a clean build and having to reparse everything anyway. But I
> could also see this class a useful to thing to always have enabled, even for personal builds, and in that case obviously I'd want to fix
> this issue.
> * When multiconfig is active, I would also like a "global" revs.inc to be generated, which is the product of aggregated the "revs.inc" for
> all the multiconfigs. Still need to think about how exactly this will work in the face of conflicting SRCREVs.
> * We have a use case for JSON format data as well ("revs.json") - I'll add that too.
> * Possibly a small tinfoil script that simply automates the task of INHERIT'ing this class, parsing all the recipes, and then dumping
> revs.inc.

Cross the first TODO off the list: https://gist.github.com/mostthingsweb/39e24b329de70c0212e5bf21a431e143/8d1a5f51b3c0bf260c730ed5716d9807c78335b8

(BTW eventually I will move this to poky-contrib; gist is just more convenient for me for the short term. If there's a desire for it to happen sooner let me know.)

Chris

Re: Looking for a way to build latest tagged releases in recipes

[chris.laplante]

> Cross the first TODO off the list:
> https://gist.github.com/mostthingsweb/39e24b329de70c0212e5bf21a431e143/8d1a5f51b3c0bf260c730ed5716d9807c78335b8
> 
> (BTW eventually I will move this to poky-contrib; gist is just more convenient for me for the short term. If there's a desire for it to
> happen sooner let me know.)

revrecord.bbclass has been moved to the cpl/revrecord branch of poky-contrib.  

I have added the ability to capture the branch for git repositories. Please check it out.

Thanks,
Chris

Re: Looking for a way to build latest tagged releases in recipes

[Pascal Huerst]

On 18/01/2020 02:11, chris.laplante@agilent.com wrote:
>> Cross the first TODO off the list:
>> https://gist.github.com/mostthingsweb/39e24b329de70c0212e5bf21a431e143/8d1a5f51b3c0bf260c730ed5716d9807c78335b8
>>
>> (BTW eventually I will move this to poky-contrib; gist is just more convenient for me for the short term. If there's a desire for it to
>> happen sooner let me know.)
> 
> revrecord.bbclass has been moved to the cpl/revrecord branch of poky-contrib.  
> 
> I have added the ability to capture the branch for git repositories. Please check it out.

Hey Chris,

thanks a lot for sharing. I will give it a try as soon as I have some time!

Regards
Pascal

Re: Looking for a way to build latest tagged releases in recipes

[chris.laplante]

> From: LAPLANTE,CHRIS (Agilent USA)
> Sent: Thursday, January 16, 2020 3:21 PM
> To: Pascal Huerst <pascal.huerst@gmail.com>; Richard Purdie <richard.purdie@linuxfoundation.org>; openembedded-
> core@lists.openembedded.org
> Cc: Rich Persaud <persaur@gmail.com>
> Subject: RE: [OE-core] Looking for a way to build latest tagged releases in recipes
> 
> > TODO:
> >
> > * Right now, the class triggers a base environment change every time, which means BitBake always reparses every recipe. I guess
> this
> > is because I'm modifying the datastore when I get bb.event.ConfigParsed and bb.event.MultiConfigParsed, in order to ensure
> > REVRECORD_DATETIME is the same across all configurations when multiconfig is active. Perhaps there is a more elegant way to do
> > this that won't trigger the environment change? To be fair I think in most cases you won't care, since I expect this class to mainly be
> > used in a continuous integration environment where you'll be doing a clean build and having to reparse everything anyway. But I
> > could also see this class a useful to thing to always have enabled, even for personal builds, and in that case obviously I'd want to fix
> > this issue.
> > * When multiconfig is active, I would also like a "global" revs.inc to be generated, which is the product of aggregated the "revs.inc"
> for
> > all the multiconfigs. Still need to think about how exactly this will work in the face of conflicting SRCREVs.
> > * We have a use case for JSON format data as well ("revs.json") - I'll add that too.
> > * Possibly a small tinfoil script that simply automates the task of INHERIT'ing this class, parsing all the recipes, and then dumping
> > revs.inc.
> 

I did some thinking about the second TODO item I had and I've decide that it is unnecessary. One revs.inc for each multiconfig is fine. To use them to reproduce a build, you simply dump the contents of each revs.inc into the corresponding multiconfig conf (or local.conf for "default"). Any algorithm I try to come up with to combine them all into a single file (for local.conf) is bound to be wrong in some way.

The third TODO is also done. The only TODO remaining is #4, a tinfoil wrapper, but so far I don't see a need. 

Chris