[Buildroot] Changes in the Buildroot autobuilders

[Buildroot] Changes in the Buildroot autobuilders

[Thomas Petazzoni]

Hello,

I've made a number of small changes in the autobuilders configuration
and setup:

 *) I've extended the mail notification script to be able to send
    per-architecture notification to persons who are specifically taking
    care of a given architecture. So far, Sonic Zhang is receiving a
    notification for Blackfin related failures, and Mischa Jonker is
    receiving a notification for ARC related failures.

 *) On the Free Electrons build server, I've modified the build script
    so that before every build, it removes 5 random files from the
    download directory. This will allow to make sure the autobuilders
    sometimes re-download tarballs or git trees, and therefore we
    should notice sooner when something is no longer available. For
    example, I've discovered that both the axel and the Blackfin
    toolchain downloads were broken, but that wasn't noticeable due to
    the tarballs already being in the download cache of the
    autobuilders.

 *) I've updated the toolchain configurations on the Free Electrons
    build server. All external toolchains built by Buildroot have been
    rebuilt using Buildroot 2013.05 (tarballs available at
    http://autobuild.buildroot.org/toolchains/tarballs/). Many
    configurations that used external toolchains have been updated to
    use the latest available versions of those toolchains and I've added
    a MIPS64 configuration that uses the Sourcery toolchain. See
    http://autobuild.buildroot.org/toolchains/configs/free-electrons/
    for the complete list of toolchains.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Changes in the Buildroot autobuilders

[Yann E. MORIN]

Thomas, All,

On 2013-06-16 18:00 +0200, Thomas Petazzoni spake thusly:
> I've made a number of small changes in the autobuilders configuration
> and setup:
> 
>  *) I've extended the mail notification script to be able to send
>     per-architecture notification to persons who are specifically taking
>     care of a given architecture. So far, Sonic Zhang is receiving a
>     notification for Blackfin related failures, and Mischa Jonker is
>     receiving a notification for ARC related failures.

It would be nice if it were also possible to somehow "subscribe" to a
list of packages and get a mail when any of these packages fails to
build.

For example, I'd like to catch errors in the packages I've submitted.

I see two options for this:
  - add a global MAINTAINERS file that list maintainers for each
    actively maintained packages (a bit like the Linux kernel does);
  - add a FOO_MAINTAINER variable to foo.mk (for each maintained 'foo'
    package).

Would that be possible?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Changes in the Buildroot autobuilders

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 16 Jun 2013 19:03:05 +0200, Yann E. MORIN wrote:

> It would be nice if it were also possible to somehow "subscribe" to a
> list of packages and get a mail when any of these packages fails to
> build.

Indeed. Are we also talking about a daily e-mail?

> For example, I'd like to catch errors in the packages I've submitted.
> 
> I see two options for this:
>   - add a global MAINTAINERS file that list maintainers for each
>     actively maintained packages (a bit like the Linux kernel does);
>   - add a FOO_MAINTAINER variable to foo.mk (for each maintained 'foo'
>     package).
> 
> Would that be possible?

Yes that would be possible. For now, I don't have a particularly strong
opinion between the two solutions you proposed. Maybe the second one is
more Buildroot-ish?

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Changes in the Buildroot autobuilders

[Yann E. MORIN]

On 2013-06-16 19:18 +0200, Thomas Petazzoni spake thusly:
> On Sun, 16 Jun 2013 19:03:05 +0200, Yann E. MORIN wrote:
> > It would be nice if it were also possible to somehow "subscribe" to a
> > list of packages and get a mail when any of these packages fails to
> > build.
> 
> Indeed. Are we also talking about a daily e-mail?

If there are build failures, yes.
Of course, no need for a mail if no build failure.

> > For example, I'd like to catch errors in the packages I've submitted.
> > 
> > I see two options for this:
> >   - add a global MAINTAINERS file that list maintainers for each
> >     actively maintained packages (a bit like the Linux kernel does);
> >   - add a FOO_MAINTAINER variable to foo.mk (for each maintained 'foo'
> >     package).
> > 
> > Would that be possible?
> 
> Yes that would be possible. For now, I don't have a particularly strong
> opinion between the two solutions you proposed. Maybe the second one is
> more Buildroot-ish?

Yes, that's my preferred one.

Should that be a single email adress, or can this be a list?
    FOO_MAINTAINERS = me at there him at overthere somone at somewhere

I would not expect more than one maintainer, but then why restrict
ourselves...

I'll adapt the manual to document this new variable, and will submit a
patch to add myself as maintainer for a few packages.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Changes in the Buildroot autobuilders

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 16 Jun 2013 19:25:08 +0200, Yann E. MORIN wrote:

> > Indeed. Are we also talking about a daily e-mail?
> 
> If there are build failures, yes.
> Of course, no need for a mail if no build failure.

Ok. This will require a little bit of work, because until now, the
autobuild.buildroot.org machine, which sends the daily e-mails, was not
using the Buildroot source code at all to do its job. So either the
slave build machines will have to submit the list of maintainers of the
failing package as part of the tarball they submit to
autobuild.buildroot.org, or autobuild.b.o will have to look at the
Buildroot source code to find the maintainers of the failing package.

Either way, it will not be immediate to implement.

> Yes, that's my preferred one.
> 
> Should that be a single email adress, or can this be a list?
>     FOO_MAINTAINERS = me at there him at overthere somone at somewhere

A list of e-mails, definitely.

> I'll adapt the manual to document this new variable, and will submit a
> patch to add myself as maintainer for a few packages.

Ok.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Changes in the Buildroot autobuilders

[Arnout Vandecappelle]

On 06/16/13 19:25, Yann E. MORIN wrote:
>>> > >For example, I'd like to catch errors in the packages I've submitted.
>>> > >
>>> > >I see two options for this:
>>> > >   - add a global MAINTAINERS file that list maintainers for each
>>> > >     actively maintained packages (a bit like the Linux kernel does);
>>> > >   - add a FOO_MAINTAINER variable to foo.mk (for each maintained 'foo'
>>> > >     package).
>>> > >
>>> > >Would that be possible?
>> >
>> >Yes that would be possible. For now, I don't have a particularly strong
>> >opinion between the two solutions you proposed. Maybe the second one is
>> >more Buildroot-ish?
> Yes, that's my preferred one.

  I don't really like to clutter the buildroot infrastructure with 
something as ephemeral as a maintainer... A more dynamic way of 
subscribing would be more relevant I think.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Changes in the Buildroot autobuilders

[Maxime Ripard]

On Sun, Jun 16, 2013 at 08:05:40PM +0200, Thomas Petazzoni wrote:
> Dear Yann E. MORIN,
> 
> On Sun, 16 Jun 2013 19:25:08 +0200, Yann E. MORIN wrote:
> 
> > > Indeed. Are we also talking about a daily e-mail?
> > 
> > If there are build failures, yes.
> > Of course, no need for a mail if no build failure.
> 
> Ok. This will require a little bit of work, because until now, the
> autobuild.buildroot.org machine, which sends the daily e-mails, was not
> using the Buildroot source code at all to do its job. So either the
> slave build machines will have to submit the list of maintainers of the
> failing package as part of the tarball they submit to
> autobuild.buildroot.org, or autobuild.b.o will have to look at the
> Buildroot source code to find the maintainers of the failing package.
> 
> Either way, it will not be immediate to implement.

Maybe having a make pkg-maintainers target that would output the list of
mails for a package maintainer would be a more reasonable thing to do.

That would allow the autobuilders to not have some knowledge of the
buildroot source code, while we remain free to change the "maintainer
implementation" at any point in time.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Changes in the Buildroot autobuilders

[Thomas Petazzoni]

Dear Maxime Ripard,

On Mon, 17 Jun 2013 09:45:35 +0200, Maxime Ripard wrote:

> > Ok. This will require a little bit of work, because until now, the
> > autobuild.buildroot.org machine, which sends the daily e-mails, was not
> > using the Buildroot source code at all to do its job. So either the
> > slave build machines will have to submit the list of maintainers of the
> > failing package as part of the tarball they submit to
> > autobuild.buildroot.org, or autobuild.b.o will have to look at the
> > Buildroot source code to find the maintainers of the failing package.
> > 
> > Either way, it will not be immediate to implement.
> 
> Maybe having a make pkg-maintainers target that would output the list of
> mails for a package maintainer would be a more reasonable thing to do.

Of course, if the solution of <pkg>_MAINTAINERS proposed by Yann is
adopted, we should have a corresponding "make show-<pkg>-maintainers"
target to display the maintainers for a given package. However...

> That would allow the autobuilders to not have some knowledge of the
> buildroot source code, while we remain free to change the "maintainer
> implementation" at any point in time.

... that doesn't solve the entire problem. For now, the slaves (doing
the builds) do not know what is the reason of the build failure. The
analysis of the log file to find out which package failed is currently
done only on the autobuild.b.o machine. But this machine doesn't have
access to the corresponding Buildroot source code to execute the "make
show-<pkg>-maintainers". So, we would have to change the logic of what
is done on the build slaves, but it's ok, it can be done. It's just not
immediate.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Changes in the Buildroot autobuilders

[Spenser Gilliland]

On Mon, Jun 17, 2013 at 2:49 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Maxime Ripard,
>
> On Mon, 17 Jun 2013 09:45:35 +0200, Maxime Ripard wrote:
>
>> > Ok. This will require a little bit of work, because until now, the
>> > autobuild.buildroot.org machine, which sends the daily e-mails, was not
>> > using the Buildroot source code at all to do its job. So either the
>> > slave build machines will have to submit the list of maintainers of the
>> > failing package as part of the tarball they submit to
>> > autobuild.buildroot.org, or autobuild.b.o will have to look at the
>> > Buildroot source code to find the maintainers of the failing package.
>> >
>> > Either way, it will not be immediate to implement.
>>
>> Maybe having a make pkg-maintainers target that would output the list of
>> mails for a package maintainer would be a more reasonable thing to do.
>
> Of course, if the solution of <pkg>_MAINTAINERS proposed by Yann is
> adopted, we should have a corresponding "make show-<pkg>-maintainers"
> target to display the maintainers for a given package. However...
>
>> That would allow the autobuilders to not have some knowledge of the
>> buildroot source code, while we remain free to change the "maintainer
>> implementation" at any point in time.
>
> ... that doesn't solve the entire problem. For now, the slaves (doing
> the builds) do not know what is the reason of the build failure. The
> analysis of the log file to find out which package failed is currently
> done only on the autobuild.b.o machine. But this machine doesn't have
> access to the corresponding Buildroot source code to execute the "make
> show-<pkg>-maintainers". So, we would have to change the logic of what
> is done on the build slaves, but it's ok, it can be done. It's just not
> immediate.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Are we sure we want to add more variables to the .mk files.  One of
Buildroot's main differentiators is the cleanliness of these files in
comparison to other packaging methods.

As an alternative, this information could be pulled from the git log
and anyone who has made a change to the package directory in the past
would be subscribed to errors.

Spenser

--
Spenser Gilliland
Computer Engineer
Doctoral Candidate

[Buildroot] Changes in the Buildroot autobuilders

[Yann E. MORIN]

Spenser, All,

On 2013-06-17 11:20 -0500, Spenser Gilliland spake thusly:
> Are we sure we want to add more variables to the .mk files.  One of
> Buildroot's main differentiators is the cleanliness of these files in
> comparison to other packaging methods.
> 
> As an alternative, this information could be pulled from the git log
> and anyone who has made a change to the package directory in the past
> would be subscribed to errors.

No, I am not in favour of this solution.

Being a 'maintainer' should be voluntary, opt-in, and explicit.

If I send an update or fix to a package does not mean I am committed to
maintain that package in the long-run. This change can be just a typo or
something minor, and might not express my interest in the wellfare of
that package.

However, if I am really interested in a package (eg. those I've
submitted, or others that are important to me), then I want to express
this intent to maintain it explicitly.

Now, I understand that adding yet more to the .mk can be seen as clutter.

The alternative to a expressing maintainership in the source would be to
have the autobuilder website offer a way to subscribe/unsubscribe to
certain conditions (eg. package, arch...). I think this is a bit
overkill, and delicate to handle.

So, there are three possibilities:

  - per-package _MAINTAINER variable in each .mk
  - global MAINTAINERS a bit like the Linux kernel
  - subscription from the autobuilder website

My preference goes to the first, but I think the second is good too.
I'd expect the third to be complex and too overkill for this, unless
Thomas (which is responsible for the autobulders ;-) ) thinks he can
handle this, of course.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Changes in the Buildroot autobuilders

[ANDY KENNEDY]

> 
> So, there are three possibilities:
> 
>   - per-package _MAINTAINER variable in each .mk
>   - global MAINTAINERS a bit like the Linux kernel
>   - subscription from the autobuilder website
> 
> My preference goes to the first, but I think the second is good too.
> I'd expect the third to be complex and too overkill for this, unless
> Thomas (which is responsible for the autobulders ;-) ) thinks he can
> handle this, of course.

I wonder if we could default to 2 if 1 is not set.  Are all packages
really owned by someone?  I would guess that there are many that just
exist and only get updated when someone needs to bump the version or
if there is a corner case.

$0.02 (USD)

Andy

[Buildroot] Changes in the Buildroot autobuilders

[Mischa Jonker]

Kennedy wrote:
>> So, there are three possibilities:
>> 
>>   - per-package _MAINTAINER variable in each .mk
>>   - global MAINTAINERS a bit like the Linux kernel
>>   - subscription from the autobuilder website
> I wonder if we could default to 2 if 1 is not set.  Are all packages
> really owned by someone?  I would guess that there are many that just
> exist and only get updated when someone needs to bump the version or
> if there is a corner case.

How would we handle architecture-specific failures for a package if we don't have a global MAINTAINERS file? Add another _MAINTAINER variable in the config.in.<arch> file?

Mischa

[Buildroot] Changes in the Buildroot autobuilders

[Spenser Gilliland]

Yann,

> No, I am not in favour of this solution.
>
> Being a 'maintainer' should be voluntary, opt-in, and explicit.

I think many people would like to know if something they did may have
triggered a bug, not just the maintainer of the package.  I'm thinking
about this from the mindset of lets let people know they broke the
build.

> If I send an update or fix to a package does not mean I am committed to
> maintain that package in the long-run. This change can be just a typo or
> something minor, and might not express my interest in the wellfare of
> that package.
>
> However, if I am really interested in a package (eg. those I've
> submitted, or others that are important to me), then I want to express
> this intent to maintain it explicitly.
>
> Now, I understand that adding yet more to the .mk can be seen as clutter.
>
> The alternative to a expressing maintainership in the source would be to
> have the autobuilder website offer a way to subscribe/unsubscribe to
> certain conditions (eg. package, arch...). I think this is a bit
> overkill, and delicate to handle.

I agree this is overkill but if it's possible it would be the best solution.

Thanks,
Spenser

--
Spenser Gilliland
Computer Engineer
Doctoral Candidate

[Buildroot] Changes in the Buildroot autobuilders

[Arnout Vandecappelle]

On 17/06/13 19:54, Yann E. MORIN wrote:
> On 2013-06-17 11:20 -0500, Spenser Gilliland spake thusly:
>> >Are we sure we want to add more variables to the .mk files.  One of
>> >Buildroot's main differentiators is the cleanliness of these files in
>> >comparison to other packaging methods.

  +1

  At least on the short term, the number of maintainers will be small and 
the number of packages per maintainer will be large. Some maintainers 
will also be fairly ephemeral - e.g. Maxime added support for systemd a 
year ago, but probably by now he doesn't consider himself a maintainer of 
it anymore. Also the rules may be a little more complex than based on 
package name.

  So I think a nice solution would be to add a script in support/scripts 
that explicitly encodes all the maintainers - maintainers can add 
themselves with patches (just like in the kernel). The a.b.o would fetch 
the script once a day just before sending out its daily mails, and call 
it with everything it knows about the build failure set in the environment.

>> >
>> >As an alternative, this information could be pulled from the git log
>> >and anyone who has made a change to the package directory in the past
>> >would be subscribed to errors.
> No, I am not in favour of this solution.
>
> Being a 'maintainer' should be voluntary, opt-in, and explicit.

  +1 too.

  However, Spenser points out in another mail:

> I think many people would like to know if something they did may have
> triggered a bug, not just the maintainer of the package.

  I could agree with sending mails automatically to the authors of a 
package commit if the commit was done in the last week or so.


  Regards,
  Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Changes in the Buildroot autobuilders

[Thomas Petazzoni]

Dear Mischa Jonker,

On Mon, 17 Jun 2013 20:09:51 +0000, Mischa Jonker wrote:

> How would we handle architecture-specific failures for a package if we don't have a global MAINTAINERS file? Add another _MAINTAINER variable in the config.in.<arch> file?

I believe architecture-specific failures would have to be handled
separately. We can't really add a maintainer into a Config.in file.

Even if we have a global MAINTAINERS file, how would architecture
maintainers be expressed? In the kernel, the MAINTAINERS file creates a
relation between a person and set of files/directories, but an
architecture in Buildroot is not a set of files or directories.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Changes in the Buildroot autobuilders

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Tue, 18 Jun 2013 08:21:17 +0200, Arnout Vandecappelle wrote:

>   So I think a nice solution would be to add a script in support/scripts 
> that explicitly encodes all the maintainers - maintainers can add 
> themselves with patches (just like in the kernel). The a.b.o would fetch 
> the script once a day just before sending out its daily mails, and call 
> it with everything it knows about the build failure set in the environment.

This does sound like a nice idea, but I would like it to also handle
architecture maintainers if possible.

So something like:

support/script/get_maintainer --arch arc --package tvheadend

would return:

Mischa Jonker <Mischa.Jonker@synopsys.com>
'Yann E. MORIN' <yann.morin.1998@free.fr>

the first being the architecture maintainer, the second being the
package maintainer.

But then, I would probably have to send one e-mail per failure, because
grouping them would be hard since there may be another failure for
tvheadend, but not on the ARC architecture.

>   However, Spenser points out in another mail:
> 
> > I think many people would like to know if something they did may have
> > triggered a bug, not just the maintainer of the package.
> 
>   I could agree with sending mails automatically to the authors of a 
> package commit if the commit was done in the last week or so.

Hum, ok. Not impossible to do, for sure.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Changes in the Buildroot autobuilders

[Yann E. MORIN]

Thomas, All,

On 2013-06-18 09:16 +0200, Thomas Petazzoni spake thusly:
> On Tue, 18 Jun 2013 08:21:17 +0200, Arnout Vandecappelle wrote:
> >   So I think a nice solution would be to add a script in support/scripts 
> > that explicitly encodes all the maintainers - maintainers can add 
> > themselves with patches (just like in the kernel). The a.b.o would fetch 
> > the script once a day just before sending out its daily mails, and call 
> > it with everything it knows about the build failure set in the environment.
> 
> This does sound like a nice idea, but I would like it to also handle
> architecture maintainers if possible.
> 
> So something like:
> 
> support/script/get_maintainer --arch arc --package tvheadend
> 
> would return:
> 
> Mischa Jonker <Mischa.Jonker@synopsys.com>
> 'Yann E. MORIN' <yann.morin.1998@free.fr>
> 
> the first being the architecture maintainer, the second being the
> package maintainer.

OK, I'll code this, then.

> But then, I would probably have to send one e-mail per failure, because
> grouping them would be hard since there may be another failure for
> tvheadend, but not on the ARC architecture.

Maybe you'd want to separate the mails:
  - one for packages
  - one for achitectures

> >   However, Spenser points out in another mail:
> > 
> > > I think many people would like to know if something they did may have
> > > triggered a bug, not just the maintainer of the package.
> > 
> >   I could agree with sending mails automatically to the authors of a 
> > package commit if the commit was done in the last week or so.
> 
> Hum, ok. Not impossible to do, for sure.

Yes, sounds like a good idea.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'