Transfering Data from GRUB to userspace?

Transfering Data from GRUB to userspace?

[Max Tottenham]

Hi Folks

I've often found myself in a position where I've needed to perform some
debugging of remote hardware that does not have a network accessible KVM
attached. To that end, being able to remotely log into a machine and
have some method of accessing GRUB2 boot time debug messages from
userspace (in my case linux), would be rather useful. It also dawned on
me that this capability might be the sort of thing that folks upstream
might be keen on.

I have done a little thinking and have thought of two possible
approaches:


1. Perform some hacking of the grub environment block (load/save env)
handling to be able to write log data to either the same file in /boot,
or perhaps a separate distinct debug log file.

2. Export the debug log to a running kernel as another initrd image. In
other words, have GRUB2 buffer a bunch of log messages and then append
that log data as a separate initrd archive. Some early userspace program
would then be able to copy that archive to the real filesystem, or
stick it into the kernel log buffer. The logfile in the
initrd that GRUB generates could have either a configurable name based
on some setting in grub.cfg, or some well known default name (like
/grub2-debug.log)


I initially started with 1, but I dislike the approach of having GRUB2
write to the filesystem, it's also a little cumbersome to setup,


Approach 2 seems like it would work (at least for linux, I've no
experience with the BSDs but presumably any OS that can load multiple
initrd's would be able to use this approach too), and has the advantage
that it doesn't have to touch any data on disk.

If there is no objection I can whip up a POC patch for 2 and post it to
the list?

Regards

-- 
Max Tottenham       | mtottenh@akamai.com
Senior Software Engineer, Server Platform Engineering
/(* Akamai Technologies

Re: Transfering Data from GRUB to userspace?

[Daniel Kiper]

Hi Max,

On Mon, Oct 28, 2019 at 10:58:21AM +0000, Max Tottenham via Grub-devel wrote:
> Hi Folks
>
> I've often found myself in a position where I've needed to perform some
> debugging of remote hardware that does not have a network accessible KVM
> attached. To that end, being able to remotely log into a machine and
> have some method of accessing GRUB2 boot time debug messages from
> userspace (in my case linux), would be rather useful. It also dawned on
> me that this capability might be the sort of thing that folks upstream
> might be keen on.
>
> I have done a little thinking and have thought of two possible
> approaches:
>
> 1. Perform some hacking of the grub environment block (load/save env)
> handling to be able to write log data to either the same file in /boot,
> or perhaps a separate distinct debug log file.

This can be difficult and unreliable.

> 2. Export the debug log to a running kernel as another initrd image. In
> other words, have GRUB2 buffer a bunch of log messages and then append
> that log data as a separate initrd archive. Some early userspace program
> would then be able to copy that archive to the real filesystem, or
> stick it into the kernel log buffer. The logfile in the
> initrd that GRUB generates could have either a configurable name based
> on some setting in grub.cfg, or some well known default name (like
> /grub2-debug.log)
>
> I initially started with 1, but I dislike the approach of having GRUB2
> write to the filesystem, it's also a little cumbersome to setup,
>
> Approach 2 seems like it would work (at least for linux, I've no
> experience with the BSDs but presumably any OS that can load multiple
> initrd's would be able to use this approach too), and has the advantage
> that it doesn't have to touch any data on disk.

Yeah, I think you are right. Additionally, please take a look at [1] and
especially at [2]. The series is about to commit and setup_indirect part
can be very useful for you. And I would like to see similar feature in the
Multiboot2 protocol [3].

Additionally, I think you can take a look at ACPI ERST and UEFI variables.
Both can be interesting for you and are used in Linux kernel by something
called pstore (in general "grep -Ir pstore Documentation" will show you
some interesting things, e.g. ramoops). Though this stuff is limited in size.

Daniel

[1] https://lkml.org/lkml/2019/10/24/378
[2] https://lkml.org/lkml/2019/10/24/387
[3] https://www.gnu.org/software/grub/manual/multiboot2/

Re: Transfering Data from GRUB to userspace?

[Max Tottenham]

On 10/28, Daniel Kiper wrote:
> Hi Max,

...

> Yeah, I think you are right. Additionally, please take a look at [1] and
> especially at [2]. The series is about to commit and setup_indirect part
> can be very useful for you. And I would like to see similar feature in the
> Multiboot2 protocol [3].

So it looks to me like setup_data is linux specific, which is fine for
me, but would mean that the BSDs wouldn't be able to take advantage of
some more generic bootloader <-> userspace interface (like a file in a
ramdisk).

I suppose for adding support to [3] the idea would be to define a new
generic boot information structure.

Maybe something like:

Generic 64bit data blob:
    +----------------------------------------------------+
u32 | type = 22                                          |
u32 | size = 28                                          |
u64 | data_load_addr = <paddr of loaded data>            |
u64 | data_len = <length of loaded data>                 |
u32 | data_type = <Some ENUM>                            |
    +----------------------------------------------------+

From reading the multiboot spec it doesn't sound like there is a
requirement that the boot information data-structure have all it's
associated structures be located in a contiguous region (unlike linux's
setup_data).

I think given the different possible approaches, I'll come up with
a formal API that could have several providers (setup_indirect v.s. initramfs
files, v.s. multiboot boot information structs), and provide an
implementation based on setup_indirect.

I think something like:

struct grub_boot_data_t;

struct grub_boot_data_t {
    grub_addr_t addr;
    uint64_t len;
    grub_uint32_t type;
    struct grub_boot_data_t* next;
};

struct grub_boot_data_provider_t {
 int (*supports) (void*);
 int (*setup) (void*, grub_boot_data_t*);
 struct grub_boot_data_provider_t *next;   
}

struct grub_boot_data_t* addl_data_head;
struct grub_boot_data_provider *boot_data_providers;

grub_boot_data_setup(void* ctx)  {
    for (p: boot_data_providers) {
        if (p->supports(ctx))
            p->setup(ctx, addl_data_head);
    }
}

grub_boot_data_register_provider() {

}

Where each backend would register a 'boot_data_provider', ctx would
contain either a pointer to the kernel setup header/multiboot header,
supports() would indicate whether this backend supports the loader used in
ctx, and setup() would perform the actual business of setting up the
correct structures etc.

> Additionally, I think you can take a look at ACPI ERST and UEFI variables.
> Both can be interesting for you and are used in Linux kernel by something
> called pstore (in general "grep -Ir pstore Documentation" will show you
> some interesting things, e.g. ramoops). Though this stuff is limited in size.

I'm weary of this approach, at least in the UEFI case I'd be weary of
writing to NVRAM backed EFI variables, as I'm pretty sure these NVRAM
chips have pretty low write limits (1-10k write cycles) and are meant to
be updated pretty infrequently. 

I wouldn't want to create an interface that developers might use
thinking that it's fine to stick stuff in there on every boot, only to
find out their NVRAM becomes a ROM after a couple of years.

> 
> Daniel
> 
> [1] https://lkml.org/lkml/2019/10/24/378
> [2] https://lkml.org/lkml/2019/10/24/387
> [3] https://www.gnu.org/software/grub/manual/multiboot2/

-- 
Max Tottenham       | mtottenh@akamai.com
Senior Software Engineer, Server Platform Engineering
/(* Akamai Technologies

Re: Transfering Data from GRUB to userspace?

[Toomas Soome]

> On 14. Nov 2019, at 12:27, Max Tottenham via Grub-devel <grub-devel@gnu.org> wrote:
> 
> On 10/28, Daniel Kiper wrote:
>> Hi Max,
> 
> ...
> 
>> Yeah, I think you are right. Additionally, please take a look at [1] and
>> especially at [2]. The series is about to commit and setup_indirect part
>> can be very useful for you. And I would like to see similar feature in the
>> Multiboot2 protocol [3].
> 
> So it looks to me like setup_data is linux specific, which is fine for
> me, but would mean that the BSDs wouldn't be able to take advantage of
> some more generic bootloader <-> userspace interface (like a file in a
> ramdisk).
> 


multiboot2 already does have modules mechanism, which is pretty generic because the identification can be done via module command line.

For example, for illumos we are providing boot loader environment variables as module (commandline) type=environment, console font as type=console-font and so on.

rgds,
toomas


> I suppose for adding support to [3] the idea would be to define a new
> generic boot information structure.
> 
> Maybe something like:
> Generic 64bit data blob:
>    +----------------------------------------------------+
> u32 | type = 22                                          |
> u32 | size = 28                                          |
> u64 | data_load_addr = <paddr of loaded data>            |
> u64 | data_len = <length of loaded data>                 |
> u32 | data_type = <Some ENUM>                            |
>    +----------------------------------------------------+
> 
> From reading the multiboot spec it doesn't sound like there is a
> requirement that the boot information data-structure have all it's
> associated structures be located in a contiguous region (unlike linux's
> setup_data).
> 
> I think given the different possible approaches, I'll come up with
> a formal API that could have several providers (setup_indirect v.s. initramfs
> files, v.s. multiboot boot information structs), and provide an
> implementation based on setup_indirect.
> 
> I think something like:
> 
> struct grub_boot_data_t;
> 
> struct grub_boot_data_t {
>    grub_addr_t addr;
>    uint64_t len;
>    grub_uint32_t type;
>    struct grub_boot_data_t* next;
> };
> 
> struct grub_boot_data_provider_t {
> int (*supports) (void*);
> int (*setup) (void*, grub_boot_data_t*);
> struct grub_boot_data_provider_t *next;   
> }
> 
> struct grub_boot_data_t* addl_data_head;
> struct grub_boot_data_provider *boot_data_providers;
> 
> grub_boot_data_setup(void* ctx)  {
>    for (p: boot_data_providers) {
>        if (p->supports(ctx))
>            p->setup(ctx, addl_data_head);
>    }
> }
> 
> grub_boot_data_register_provider() {
> 
> }
> 
> Where each backend would register a 'boot_data_provider', ctx would
> contain either a pointer to the kernel setup header/multiboot header,
> supports() would indicate whether this backend supports the loader used in
> ctx, and setup() would perform the actual business of setting up the
> correct structures etc.
> 
>> Additionally, I think you can take a look at ACPI ERST and UEFI variables.
>> Both can be interesting for you and are used in Linux kernel by something
>> called pstore (in general "grep -Ir pstore Documentation" will show you
>> some interesting things, e.g. ramoops). Though this stuff is limited in size.
> 
> I'm weary of this approach, at least in the UEFI case I'd be weary of
> writing to NVRAM backed EFI variables, as I'm pretty sure these NVRAM
> chips have pretty low write limits (1-10k write cycles) and are meant to
> be updated pretty infrequently. 
> 
> I wouldn't want to create an interface that developers might use
> thinking that it's fine to stick stuff in there on every boot, only to
> find out their NVRAM becomes a ROM after a couple of years.
> 
>> 
>> Daniel
>> 
>> [1] https://lkml.org/lkml/2019/10/24/378
>> [2] https://lkml.org/lkml/2019/10/24/387
>> [3] https://www.gnu.org/software/grub/manual/multiboot2/
> 
> -- 
> Max Tottenham       | mtottenh@akamai.com
> Senior Software Engineer, Server Platform Engineering
> /(* Akamai Technologies
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel

Re: Transfering Data from GRUB to userspace?

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

Did you look at cpio generation facility in initrd command? This may be
what your are looking for

On Mon, 28 Oct 2019, 03:59 Max Tottenham via Grub-devel, <grub-devel@gnu.org>
wrote:

> Hi Folks
>
> I've often found myself in a position where I've needed to perform some
> debugging of remote hardware that does not have a network accessible KVM
> attached. To that end, being able to remotely log into a machine and
> have some method of accessing GRUB2 boot time debug messages from
> userspace (in my case linux), would be rather useful. It also dawned on
> me that this capability might be the sort of thing that folks upstream
> might be keen on.
>
> I have done a little thinking and have thought of two possible
> approaches:
>
>
> 1. Perform some hacking of the grub environment block (load/save env)
> handling to be able to write log data to either the same file in /boot,
> or perhaps a separate distinct debug log file.
>
> 2. Export the debug log to a running kernel as another initrd image. In
> other words, have GRUB2 buffer a bunch of log messages and then append
> that log data as a separate initrd archive. Some early userspace program
> would then be able to copy that archive to the real filesystem, or
> stick it into the kernel log buffer. The logfile in the
> initrd that GRUB generates could have either a configurable name based
> on some setting in grub.cfg, or some well known default name (like
> /grub2-debug.log)
>
>
> I initially started with 1, but I dislike the approach of having GRUB2
> write to the filesystem, it's also a little cumbersome to setup,
>
>
> Approach 2 seems like it would work (at least for linux, I've no
> experience with the BSDs but presumably any OS that can load multiple
> initrd's would be able to use this approach too), and has the advantage
> that it doesn't have to touch any data on disk.
>
> If there is no objection I can whip up a POC patch for 2 and post it to
> the list?
>
> Regards
>
> --
> Max Tottenham       | mtottenh@akamai.com
> Senior Software Engineer, Server Platform Engineering
> /(* Akamai Technologies
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 3017 bytes --]

Re: Transfering Data from GRUB to userspace?

[Max Tottenham]

On 11/14, Vladimir 'phcoder' Serbinenko wrote:
> Did you look at cpio generation facility in initrd command? This may be
> what your are looking for
> 

I'm assuming you mean the various grub_initrd_load()/grub_initrd_close()
functions? I couldn't see any actual grub commands that provide cpio
generation.

-- 
Max Tottenham       | mtottenh@akamai.com
Senior Software Engineer, Server Platform Engineering
/(* Akamai Technologies

Re: Transfering Data from GRUB to userspace?

[Hans de Goede]

Hi,

<snip>

Sorry for jumping in mid thread, but I just noticed this and some functionality
like this has my interest too. I especially triggered on the cpio / initrd
proposal.

I have been discussing something similar with the systemd folks, see
some background see:

https://lists.freedesktop.org/archives/systemd-devel/2019-September/043498.html
https://lists.freedesktop.org/archives/systemd-devel/2019-September/043515.html
https://lists.freedesktop.org/archives/systemd-devel/2019-September/043520.html
https://lists.freedesktop.org/archives/systemd-devel/2019-September/043521.html

Where the last one goed into the cpio / appended initrd concept.

What we were discussing there is a way to pass the kbd-layout to the bootloader
and then having the bootloader pass it back to the initrd (so userspace in
a way).

The problem is that localectl updates /etc/vconsole.conf but that does not
influence that file in existing initrds so the initrd has a wrong value
after changing the kbd layout and regenerating the initrd is undesirable,
since we keep old kernel + initrds around as a fallback in case something
breaks and if it is the initrd which broke then regenerating it will
break the fallback.

So Lennart said that we should really also pass the kbdmap to the bootloader
in case the bootloader supports it (for the cmdline mode). Lennart suggested
dropping a kbdmap file next to say grubenv which contains the kbdmap info.
I instead am suggesting a "config" file which contains
key-value pairs to make the mechanism more generic.

Now the problem is, we still need to get this info to the initrd. Typically
the initrd does not mount /boot or the ESP, so it cannot directly get
to the kbdmap or config file. Lennart suggested simply passing it on
the kernel cmdline which will work for the kbdmap case, but again is not
very generic.

I instead suggested making the "config" (or whatever we are going to name ir)
file available to the initrd variant by append an on the fly generated cpio
archive to the initrd with that file in it. Which is why I triggered on the
mentioning of using cpio to pass info from the bootloader to userspace.

Please keep in mind that what I'm looking for is a bootloader agnostic
solution. Requiring changes to the bootloader is fine, but it is something
which should not be grub specific.

Anyways TL;DR: I think that being able to pass an extra on the fly
generated cpio archive appended to the initrd when booting is something
which would be useful to have.

Regards,

Hans

Re: Transfering Data from GRUB to userspace?

[Daniel Kiper]

On Thu, Nov 14, 2019 at 10:27:10AM +0000, Max Tottenham via Grub-devel wrote:
> On 10/28, Daniel Kiper wrote:
> > Hi Max,
>
> ...
>
> > Yeah, I think you are right. Additionally, please take a look at [1] and
> > especially at [2]. The series is about to commit and setup_indirect part

FYI, it is in tip:x86/boot right now and will land in 5.5.

> > can be very useful for you. And I would like to see similar feature in the
> > Multiboot2 protocol [3].
>
> So it looks to me like setup_data is linux specific, which is fine for
> me, but would mean that the BSDs wouldn't be able to take advantage of
> some more generic bootloader <-> userspace interface (like a file in a
> ramdisk).
>
> I suppose for adding support to [3] the idea would be to define a new
> generic boot information structure.
>
> Maybe something like:
>
> Generic 64bit data blob:
>     +----------------------------------------------------+
> u32 | type = 22                                          |
> u32 | size = 28                                          |
> u64 | data_load_addr = <paddr of loaded data>            |
> u64 | data_len = <length of loaded data>                 |
> u32 | data_type = <Some ENUM>                            |

Why do you need data_type?

>     +----------------------------------------------------+
>
> From reading the multiboot spec it doesn't sound like there is a
> requirement that the boot information data-structure have all it's
> associated structures be located in a contiguous region (unlike linux's
> setup_data).

Yes, there is no such requirement.

> I think given the different possible approaches, I'll come up with
> a formal API that could have several providers (setup_indirect v.s. initramfs
> files, v.s. multiboot boot information structs), and provide an
> implementation based on setup_indirect.
>
> I think something like:
>
> struct grub_boot_data_t;
>
> struct grub_boot_data_t {
>     grub_addr_t addr;
>     uint64_t len;
>     grub_uint32_t type;
>     struct grub_boot_data_t* next;
> };
>
> struct grub_boot_data_provider_t {
>  int (*supports) (void*);
>  int (*setup) (void*, grub_boot_data_t*);
>  struct grub_boot_data_provider_t *next;
> }
>
> struct grub_boot_data_t* addl_data_head;
> struct grub_boot_data_provider *boot_data_providers;
>
> grub_boot_data_setup(void* ctx)  {
>     for (p: boot_data_providers) {
>         if (p->supports(ctx))
>             p->setup(ctx, addl_data_head);
>     }
> }
>
> grub_boot_data_register_provider() {
>
> }
>
> Where each backend would register a 'boot_data_provider', ctx would
> contain either a pointer to the kernel setup header/multiboot header,
> supports() would indicate whether this backend supports the loader used in
> ctx, and setup() would perform the actual business of setting up the
> correct structures etc.

So, AIUI you want to have a self-contained "log" structure which can be
pointed from setup_indirect or Multiboot2. Am I right? If this is true
then I am OK with the idea. However, I would like to ask you to involve
other interested groups and people in further discussions, e.g. relevant
Linux maintainers, BSD maintainers and Xen maintainers (yes, Xen is using
Multiboot2). If you need some email addresses I can send you them.

> > Additionally, I think you can take a look at ACPI ERST and UEFI variables.
> > Both can be interesting for you and are used in Linux kernel by something
> > called pstore (in general "grep -Ir pstore Documentation" will show you
> > some interesting things, e.g. ramoops). Though this stuff is limited in size.
>
> I'm weary of this approach, at least in the UEFI case I'd be weary of
> writing to NVRAM backed EFI variables, as I'm pretty sure these NVRAM
> chips have pretty low write limits (1-10k write cycles) and are meant to
> be updated pretty infrequently.

Yeah...

> I wouldn't want to create an interface that developers might use
> thinking that it's fine to stick stuff in there on every boot, only to
> find out their NVRAM becomes a ROM after a couple of years.

I fully agree.

Daniel

Re: Transfering Data from GRUB to userspace?

[Daniel Kiper]

On Thu, Nov 14, 2019 at 12:56:25PM +0200, Toomas Soome via Grub-devel wrote:
> > On 14. Nov 2019, at 12:27, Max Tottenham via Grub-devel <grub-devel@gnu.org> wrote:
> > On 10/28, Daniel Kiper wrote:
> >> Hi Max,
> >
> > ...
> >
> >> Yeah, I think you are right. Additionally, please take a look at [1] and
> >> especially at [2]. The series is about to commit and setup_indirect part
> >> can be very useful for you. And I would like to see similar feature in the
> >> Multiboot2 protocol [3].
> >
> > So it looks to me like setup_data is linux specific, which is fine for
> > me, but would mean that the BSDs wouldn't be able to take advantage of
> > some more generic bootloader <-> userspace interface (like a file in a
> > ramdisk).
>
> multiboot2 already does have modules mechanism, which is pretty
> generic because the identification can be done via module command
> line.
>
> For example, for illumos we are providing boot loader environment
> variables as module (commandline) type=environment, console font as
> type=console-font and so on.

This is another option but I prefer to have separate Multiboot2 type for
"log" thing. This way we will not confuse existing users of module
Multiboot2 type.

Daniel

Re: Transfering Data from GRUB to userspace?

[Daniel Kiper]

On Tue, Nov 19, 2019 at 02:01:11AM +0000, Tottenham, Max via Grub-devel wrote:
> Original mail seems to have been eaten again, apologies if you already received this.
>
> On 11/15, Daniel Kiper wrote:
> > On Thu, Nov 14, 2019 at 10:27:10AM +0000, Max Tottenham via Grub-devel wrote:
> > > On 10/28, Daniel Kiper wrote:
> > > > Hi Max,
> > >
> > > ...
> > > I suppose for adding support to [3] the idea would be to define a new
> > > generic boot information structure.
> > >
> > > Maybe something like:
> > >
> > > Generic 64bit data blob:
> > >     +----------------------------------------------------+
> > > u32 | type = 22                                          |
> > > u32 | size = 28                                          |
> > > u64 | data_load_addr = <paddr of loaded data>            |
> > > u64 | data_len = <length of loaded data>                 |
> > > u32 | data_type = <Some ENUM>                            |
> >
> > Why do you need data_type?
> >
> > >     +----------------------------------------------------+
>
> My initial thought was that it would be an easy way to create a
> 'generic' boot info struct where 'type' defines that it's a generic
> type, and 'data_type' is a separate type field that is interpreted by
> defined by the loaded OS, I suppose the alternative would be to declare

This is more or less what I expected. I do not want that because it will
make confusion and may encourage people to interpret data_type as they
want to without obeying the spec.

> separate boot info struct's for each purpose (so for my purpose a simple
> 'log' type struct would suffice).

I am OK with that. And this log may define various log types and formats
in own header.

> > So, AIUI you want to have a self-contained "log" structure which can be
> > pointed from setup_indirect or Multiboot2. Am I right? If this is true
> > then I am OK with the idea.
>
> Correct.
>
> > However, I would like to ask you to involve other interested groups
> > and people in further discussions, e.g. relevant Linux maintainers,
> > BSD maintainers and Xen maintainers (yes, Xen is using Multiboot2). If
> > you need some email addresses I can send you them.
> >
>
> I'd greatly appreciate it if you'd be able to send me over the relevant
> maintainers/mailing list addresses.

Will do.

Daniel

Re: Transfering Data from GRUB to userspace?

[Tottenham, Max]

Original mail seems to have been eaten again, apologies if you already received this. 

On 11/15, Daniel Kiper wrote:
> On Thu, Nov 14, 2019 at 10:27:10AM +0000, Max Tottenham via Grub-devel wrote:
> > On 10/28, Daniel Kiper wrote:
> > > Hi Max,
> >
> > ...
> > I suppose for adding support to [3] the idea would be to define a new
> > generic boot information structure.
> >
> > Maybe something like:
> >
> > Generic 64bit data blob:
> >     +----------------------------------------------------+
> > u32 | type = 22                                          |
> > u32 | size = 28                                          |
> > u64 | data_load_addr = <paddr of loaded data>            |
> > u64 | data_len = <length of loaded data>                 |
> > u32 | data_type = <Some ENUM>                            |
> 
> Why do you need data_type?
> 
> >     +----------------------------------------------------+

My initial thought was that it would be an easy way to create a
'generic' boot info struct where 'type' defines that it's a generic
type, and 'data_type' is a separate type field that is interpreted by
defined by the loaded OS, I suppose the alternative would be to declare
separate boot info struct's for each purpose (so for my purpose a simple
'log' type struct would suffice).

> 
> So, AIUI you want to have a self-contained "log" structure which can be
> pointed from setup_indirect or Multiboot2. Am I right? If this is true
> then I am OK with the idea. 

Correct.

> However, I would like to ask you to involve other interested groups
> and people in further discussions, e.g. relevant Linux maintainers,
> BSD maintainers and Xen maintainers (yes, Xen is using Multiboot2). If
> you need some email addresses I can send you them.
> 

I'd greatly appreciate it if you'd be able to send me over the relevant
maintainers/mailing list addresses.


Thanks!

-- 
Max Tottenham       | mtottenh@akamai.com
Senior Software Engineer, Server Platform Engineering
/(* Akamai Technologies