* CI to stop testing meta-* layers not in tested machine @ 2019-03-14 13:38 Andrew Geissler 2019-03-14 17:07 ` Patrick Venture ` (2 more replies) 0 siblings, 3 replies; 13+ messages in thread From: Andrew Geissler @ 2019-03-14 13:38 UTC (permalink / raw) To: OpenBMC Maillist I took an action item from last weeks Infrastructure Workgroup. The point was we're wasting CI resources by testing meta-* commits that are not actually tested by any of the machines in the CI job. We're also falsely marking those commits as Verified because if they are not in any of the systems under test, they're not being tested at all. The systems currently run as a part of the meta-* CI jobs are here: https://openpower.xyz/view/CI/job/run-meta-ci/ Some quick grepping indicates the following meta-* repos are not being tested: meta-arm meta-evb meta-google meta-hxt meta-inspur meta-intel meta-inventec meta-mellanox meta-nuvoton meta-portwell meta-qualcomm meta-quanta meta-raspberrypi meta-security meta-x86 meta-xilinx This would mean the maintainers of the above repos would need to +1 Verify the changes to these layers before merging. Are there any advantages to running CI against meta-* layers that are not in a machine being built? Are there other machines we can add to CI that would cover some of the meta layers above? The general criteria for getting a machine added to CI is that it's actively being developed and supported. We also need to balance our CI compute resources so the overall goal (in my mind) would be to pick the machines that cover the most meta layers. Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-03-14 13:38 CI to stop testing meta-* layers not in tested machine Andrew Geissler @ 2019-03-14 17:07 ` Patrick Venture 2019-05-09 19:36 ` Andrew Geissler 2019-05-10 1:10 ` Joel Stanley 2 siblings, 0 replies; 13+ messages in thread From: Patrick Venture @ 2019-03-14 17:07 UTC (permalink / raw) To: Andrew Geissler; +Cc: OpenBMC Maillist On Thu, Mar 14, 2019 at 6:39 AM Andrew Geissler <geissonator@gmail.com> wrote: > > I took an action item from last weeks Infrastructure Workgroup. > > The point was we're wasting CI resources by testing meta-* > commits that are not actually tested by any of the machines in the > CI job. We're also falsely marking those commits as Verified because > if they are not in any of the systems under test, they're not being > tested at all. > > The systems currently run as a part of the meta-* CI jobs are here: > https://openpower.xyz/view/CI/job/run-meta-ci/ > > Some quick grepping indicates the following meta-* repos are not > being tested: > > meta-arm > meta-evb > meta-google > meta-hxt > meta-inspur > meta-intel > meta-inventec > meta-mellanox > meta-nuvoton > meta-portwell > meta-qualcomm > meta-quanta > meta-raspberrypi > meta-security > meta-x86 > meta-xilinx > > This would mean the maintainers of the above repos would need to +1 > Verify the changes to these layers before merging. > > Are there any advantages to running CI against meta-* layers that > are not in a machine being built? Are there other machines we can > add to CI that would cover some of the meta layers above? The > general criteria for getting a machine added to CI is that it's actively > being developed and supported. We also need to balance our > CI compute resources so the overall goal (in my mind) would be > to pick the machines that cover the most meta layers. @OCP Summit today talking to Brad. It's possible I'll end up pushing up a machine configuration that'll use meta-google. However, as far as not testing it during a build, I think yeah, with limited compute resources, there's only so much we can do. > > Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-03-14 13:38 CI to stop testing meta-* layers not in tested machine Andrew Geissler 2019-03-14 17:07 ` Patrick Venture @ 2019-05-09 19:36 ` Andrew Geissler 2019-05-10 1:10 ` Joel Stanley 2 siblings, 0 replies; 13+ messages in thread From: Andrew Geissler @ 2019-05-09 19:36 UTC (permalink / raw) To: OpenBMC Maillist I'd like to revisit this topic. It looks like the meta-* layers I would remove from meta-CI are still the same as below. We've added the meta-facebook/meta-tiogapass since my last email to CI so that will be a meta layer (meta-facebook) we continue to test. Please let me know if you have any questions or concerns. Thanks, Andrew On Thu, Mar 14, 2019 at 8:38 AM Andrew Geissler <geissonator@gmail.com> wrote: > > I took an action item from last weeks Infrastructure Workgroup. > > The point was we're wasting CI resources by testing meta-* > commits that are not actually tested by any of the machines in the > CI job. We're also falsely marking those commits as Verified because > if they are not in any of the systems under test, they're not being > tested at all. > > The systems currently run as a part of the meta-* CI jobs are here: > https://openpower.xyz/view/CI/job/run-meta-ci/ > > Some quick grepping indicates the following meta-* repos are not > being tested: > > meta-arm > meta-evb > meta-google > meta-hxt > meta-inspur > meta-intel > meta-inventec > meta-mellanox > meta-nuvoton > meta-portwell > meta-qualcomm > meta-quanta > meta-raspberrypi > meta-security > meta-x86 > meta-xilinx > > This would mean the maintainers of the above repos would need to +1 > Verify the changes to these layers before merging. > > Are there any advantages to running CI against meta-* layers that > are not in a machine being built? Are there other machines we can > add to CI that would cover some of the meta layers above? The > general criteria for getting a machine added to CI is that it's actively > being developed and supported. We also need to balance our > CI compute resources so the overall goal (in my mind) would be > to pick the machines that cover the most meta layers. > > Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-03-14 13:38 CI to stop testing meta-* layers not in tested machine Andrew Geissler 2019-03-14 17:07 ` Patrick Venture 2019-05-09 19:36 ` Andrew Geissler @ 2019-05-10 1:10 ` Joel Stanley 2019-05-10 14:33 ` Patrick Venture 2 siblings, 1 reply; 13+ messages in thread From: Joel Stanley @ 2019-05-10 1:10 UTC (permalink / raw) To: Andrew Geissler, Benjamin Fair; +Cc: OpenBMC Maillist On Thu, 14 Mar 2019 at 13:39, Andrew Geissler <geissonator@gmail.com> wrote: > > I took an action item from last weeks Infrastructure Workgroup. > > The point was we're wasting CI resources by testing meta-* > commits that are not actually tested by any of the machines in the > CI job. We're also falsely marking those commits as Verified because > if they are not in any of the systems under test, they're not being > tested at all. > > The systems currently run as a part of the meta-* CI jobs are here: > https://openpower.xyz/view/CI/job/run-meta-ci/ > Are there any advantages to running CI against meta-* layers that > are not in a machine being built? Are there other machines we can > add to CI that would cover some of the meta layers above? The > general criteria for getting a machine added to CI is that it's actively > being developed and supported. We also need to balance our > CI compute resources so the overall goal (in my mind) would be > to pick the machines that cover the most meta layers. I'd like to have a nuvoton based machine so we have some confidence that kernel bumps aren't broken. That would mean adding the evb-nuvoton or gsj machines to CI. Cheers, Joel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-05-10 1:10 ` Joel Stanley @ 2019-05-10 14:33 ` Patrick Venture 2019-05-10 18:45 ` Andrew Geissler 0 siblings, 1 reply; 13+ messages in thread From: Patrick Venture @ 2019-05-10 14:33 UTC (permalink / raw) To: Joel Stanley; +Cc: Andrew Geissler, Benjamin Fair, OpenBMC Maillist From: Joel Stanley <joel@jms.id.au> Date: Thu, May 9, 2019 at 6:11 PM To: Andrew Geissler, Benjamin Fair Cc: OpenBMC Maillist > On Thu, 14 Mar 2019 at 13:39, Andrew Geissler <geissonator@gmail.com> wrote: > > > > I took an action item from last weeks Infrastructure Workgroup. > > > > The point was we're wasting CI resources by testing meta-* > > commits that are not actually tested by any of the machines in the > > CI job. We're also falsely marking those commits as Verified because > > if they are not in any of the systems under test, they're not being > > tested at all. > > > > The systems currently run as a part of the meta-* CI jobs are here: > > https://openpower.xyz/view/CI/job/run-meta-ci/ > > > Are there any advantages to running CI against meta-* layers that > > are not in a machine being built? Are there other machines we can > > add to CI that would cover some of the meta layers above? The > > general criteria for getting a machine added to CI is that it's actively > > being developed and supported. We also need to balance our > > CI compute resources so the overall goal (in my mind) would be > > to pick the machines that cover the most meta layers. > > I'd like to have a nuvoton based machine so we have some confidence > that kernel bumps aren't broken. > > That would mean adding the evb-nuvoton or gsj machines to CI. I vote for the gsj machine. Not that it's a democracy :) > > Cheers, > > Joel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-05-10 14:33 ` Patrick Venture @ 2019-05-10 18:45 ` Andrew Geissler 2019-06-14 4:54 ` Joel Stanley 0 siblings, 1 reply; 13+ messages in thread From: Andrew Geissler @ 2019-05-10 18:45 UTC (permalink / raw) To: Patrick Venture; +Cc: Joel Stanley, Benjamin Fair, OpenBMC Maillist On Fri, May 10, 2019 at 9:33 AM Patrick Venture <venture@google.com> wrote: > > From: Joel Stanley <joel@jms.id.au> > Date: Thu, May 9, 2019 at 6:11 PM > To: Andrew Geissler, Benjamin Fair > Cc: OpenBMC Maillist > > > On Thu, 14 Mar 2019 at 13:39, Andrew Geissler <geissonator@gmail.com> wrote: > > > > > > I took an action item from last weeks Infrastructure Workgroup. > > > > > > The point was we're wasting CI resources by testing meta-* > > > commits that are not actually tested by any of the machines in the > > > CI job. We're also falsely marking those commits as Verified because > > > if they are not in any of the systems under test, they're not being > > > tested at all. > > > > > > The systems currently run as a part of the meta-* CI jobs are here: > > > https://openpower.xyz/view/CI/job/run-meta-ci/ > > > > > Are there any advantages to running CI against meta-* layers that > > > are not in a machine being built? Are there other machines we can > > > add to CI that would cover some of the meta layers above? The > > > general criteria for getting a machine added to CI is that it's actively > > > being developed and supported. We also need to balance our > > > CI compute resources so the overall goal (in my mind) would be > > > to pick the machines that cover the most meta layers. > > > > I'd like to have a nuvoton based machine so we have some confidence > > that kernel bumps aren't broken. > > > > That would mean adding the evb-nuvoton or gsj machines to CI. > > I vote for the gsj machine. Not that it's a democracy :) I gave this a try but ran into https://github.com/openbmc/openbmc/issues/3542 Be great to get gsj into CI since it would give us a few new layers for coverage. google provides a good chunk of the CI build infrastructure for OpenBMC so you definitely get a vote :) Andrew > > > > > Cheers, > > > > Joel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-05-10 18:45 ` Andrew Geissler @ 2019-06-14 4:54 ` Joel Stanley 2019-06-14 15:55 ` Benjamin Fair 0 siblings, 1 reply; 13+ messages in thread From: Joel Stanley @ 2019-06-14 4:54 UTC (permalink / raw) To: OpenBMC Maillist, Patrick Venture, Benjamin Fair On Fri, 10 May 2019 at 18:46, Andrew Geissler <geissonator@gmail.com> wrote: > > On Fri, May 10, 2019 at 9:33 AM Patrick Venture <venture@google.com> wrote: > > > I'd like to have a nuvoton based machine so we have some confidence > > > that kernel bumps aren't broken. > > > > > > That would mean adding the evb-nuvoton or gsj machines to CI. > > > > I vote for the gsj machine. Not that it's a democracy :) > > I gave this a try but ran into https://github.com/openbmc/openbmc/issues/3542 > Be great to get gsj into CI since it would give us a few new layers > for coverage. gsj is now supported in the kernel. Andrew tried to build the machine and ran into u-boot issues which is still blocking the machine's addition to our CI. Patrick, are you able to look into that? https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892 Once we get the u-boot issue sorted out, I propose the following changes: - drop qemu from CI. 'qemu' is actually testing on a generic arm machine. A few of us at IBM have a side project that has resulted in a high quality Qemu support for the aspeed boards, so if you would like to test in qemu I recommend grabbing palmetto or romulus and doing that. So consider this dropping the generic qemu image and instead focusing on the aspeed one. - add gsj. This gives us coverage of the nuvoton kernel and u-boot, as well as the nuvoton specific layers - add swift. This is an ast2500-based system that we're looking to use emmc flash with, and having testing for those images will be useful Cheers, Joel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-06-14 4:54 ` Joel Stanley @ 2019-06-14 15:55 ` Benjamin Fair 2019-06-17 3:29 ` Andrew Jeffery 2019-06-21 0:38 ` Joel Stanley 0 siblings, 2 replies; 13+ messages in thread From: Benjamin Fair @ 2019-06-14 15:55 UTC (permalink / raw) To: Joel Stanley; +Cc: OpenBMC Maillist, Patrick Venture, Andrew Geissler On Thu, Jun 13, 2019 at 9:54 PM Joel Stanley <joel@jms.id.au> wrote: > > On Fri, 10 May 2019 at 18:46, Andrew Geissler <geissonator@gmail.com> wrote: > > > > On Fri, May 10, 2019 at 9:33 AM Patrick Venture <venture@google.com> wrote: > > > > I'd like to have a nuvoton based machine so we have some confidence > > > > that kernel bumps aren't broken. > > > > > > > > That would mean adding the evb-nuvoton or gsj machines to CI. > > > > > > I vote for the gsj machine. Not that it's a democracy :) > > > > I gave this a try but ran into https://github.com/openbmc/openbmc/issues/3542 > > Be great to get gsj into CI since it would give us a few new layers > > for coverage. > > gsj is now supported in the kernel. > > Andrew tried to build the machine and ran into u-boot issues which is > still blocking the machine's addition to our CI. Patrick, are you able > to look into that? > > https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892 That issue will be resolved by switching to a 2019-based U-Boot branch: https://gerrit.openbmc-project.xyz/c/openbmc/meta-nuvoton/+/22556 > > > > Once we get the u-boot issue sorted out, I propose the following changes: > > - drop qemu from CI. 'qemu' is actually testing on a generic arm > machine. A few of us at IBM have a side project that has resulted in a > high quality Qemu support for the aspeed boards, so if you would like > to test in qemu I recommend grabbing palmetto or romulus and doing > that. So consider this dropping the generic qemu image and instead > focusing on the aspeed one. +1 Many things are already broken on QEMU, including phosphor-ipmi-host. It's not a useful platform to test with. > > - add gsj. This gives us coverage of the nuvoton kernel and u-boot, > as well as the nuvoton specific layers Agreed. I'd also like to switch to (or add) a Nuvoton system with a host once such a machine is supported upstream. The gsj is only a storage tray so it doesn't test IPMI bridges, power control, etc. > > - add swift. This is an ast2500-based system that we're looking to > use emmc flash with, and having testing for those images will be > useful > > Cheers, > > Joel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-06-14 15:55 ` Benjamin Fair @ 2019-06-17 3:29 ` Andrew Jeffery 2019-06-19 0:45 ` Benjamin Fair 2019-06-21 0:38 ` Joel Stanley 1 sibling, 1 reply; 13+ messages in thread From: Andrew Jeffery @ 2019-06-17 3:29 UTC (permalink / raw) To: Benjamin Fair, Joel Stanley; +Cc: Patrick Venture, OpenBMC Maillist > > > > > > > > Once we get the u-boot issue sorted out, I propose the following changes: > > > > - drop qemu from CI. 'qemu' is actually testing on a generic arm > > machine. A few of us at IBM have a side project that has resulted in a > > high quality Qemu support for the aspeed boards, so if you would like > > to test in qemu I recommend grabbing palmetto or romulus and doing > > that. So consider this dropping the generic qemu image and instead > > focusing on the aspeed one. > > +1 > > Many things are already broken on QEMU, including phosphor-ipmi-host. > It's not a useful platform to test with. > Is that a general statement about QEMU, or are you talking about the generic qemu image as Joel was? It would be great if we had more contributions on the QEMU side. There's a list of things that can be attacked in the wiki: https://github.com/openbmc/qemu/wiki Modelling I2C peripherals is usually a good first step. Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-06-17 3:29 ` Andrew Jeffery @ 2019-06-19 0:45 ` Benjamin Fair 2019-06-19 1:06 ` Andrew Jeffery 0 siblings, 1 reply; 13+ messages in thread From: Benjamin Fair @ 2019-06-19 0:45 UTC (permalink / raw) To: Andrew Jeffery; +Cc: Joel Stanley, Patrick Venture, OpenBMC Maillist On Sun, Jun 16, 2019 at 8:30 PM Andrew Jeffery <andrew@aj.id.au> wrote: > > > > > > > > > > > > > Once we get the u-boot issue sorted out, I propose the following changes: > > > > > > - drop qemu from CI. 'qemu' is actually testing on a generic arm > > > machine. A few of us at IBM have a side project that has resulted in a > > > high quality Qemu support for the aspeed boards, so if you would like > > > to test in qemu I recommend grabbing palmetto or romulus and doing > > > that. So consider this dropping the generic qemu image and instead > > > focusing on the aspeed one. > > > > +1 > > > > Many things are already broken on QEMU, including phosphor-ipmi-host. > > It's not a useful platform to test with. > > > > Is that a general statement about QEMU, or are you talking about the generic > qemu image as Joel was? That's specifically an issue with the generic QEMU image. It's because the generic image doesn't include u-boot, which is a dependency of phosphor-ipmi-host (transitively through clear-once). > > It would be great if we had more contributions on the QEMU side. There's a > list of things that can be attacked in the wiki: > > https://github.com/openbmc/qemu/wiki > > Modelling I2C peripherals is usually a good first step. > > Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-06-19 0:45 ` Benjamin Fair @ 2019-06-19 1:06 ` Andrew Jeffery 0 siblings, 0 replies; 13+ messages in thread From: Andrew Jeffery @ 2019-06-19 1:06 UTC (permalink / raw) To: Benjamin Fair; +Cc: Joel Stanley, Patrick Venture, OpenBMC Maillist On Wed, 19 Jun 2019, at 10:16, Benjamin Fair wrote: > On Sun, Jun 16, 2019 at 8:30 PM Andrew Jeffery <andrew@aj.id.au> wrote: > > > > > > > > > > > > > > > > > > Once we get the u-boot issue sorted out, I propose the following changes: > > > > > > > > - drop qemu from CI. 'qemu' is actually testing on a generic arm > > > > machine. A few of us at IBM have a side project that has resulted in a > > > > high quality Qemu support for the aspeed boards, so if you would like > > > > to test in qemu I recommend grabbing palmetto or romulus and doing > > > > that. So consider this dropping the generic qemu image and instead > > > > focusing on the aspeed one. > > > > > > +1 > > > > > > Many things are already broken on QEMU, including phosphor-ipmi-host. > > > It's not a useful platform to test with. > > > > > > > Is that a general statement about QEMU, or are you talking about the generic > > qemu image as Joel was? > > That's specifically an issue with the generic QEMU image. It's because > the generic image doesn't include u-boot, which is a dependency of > phosphor-ipmi-host (transitively through clear-once). > I've hit that before, and it's not an intuitive problem. I'm all for ditching the generic qemu machine. Back when I wrote the original aspeed support for qemu I was looking at integrating the machine configuration into the openbmc tree, but it was a little awkward at the time. It would be good if someone could pick that up, as upstream have rearranged how they invoke qemu and it should be more flexible now. Andrew ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-06-14 15:55 ` Benjamin Fair 2019-06-17 3:29 ` Andrew Jeffery @ 2019-06-21 0:38 ` Joel Stanley 2019-06-21 18:51 ` Andrew Geissler 1 sibling, 1 reply; 13+ messages in thread From: Joel Stanley @ 2019-06-21 0:38 UTC (permalink / raw) To: Benjamin Fair; +Cc: OpenBMC Maillist, Patrick Venture, Andrew Geissler On Fri, 14 Jun 2019 at 15:55, Benjamin Fair <benjaminfair@google.com> wrote: > > > > Andrew tried to build the machine and ran into u-boot issues which is > > still blocking the machine's addition to our CI. Patrick, are you able > > to look into that? > > > > https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892 > > That issue will be resolved by switching to a 2019-based U-Boot branch: > > https://gerrit.openbmc-project.xyz/c/openbmc/meta-nuvoton/+/22556 This has been merged now. Andrew G, are we able to turn on the CI? I think we have consensus to drop qemu, and enable gsj. There were no objections to enabling swift too. Cheers, Joel > > > > > > > > > Once we get the u-boot issue sorted out, I propose the following changes: > > > > - drop qemu from CI. 'qemu' is actually testing on a generic arm > > machine. A few of us at IBM have a side project that has resulted in a > > high quality Qemu support for the aspeed boards, so if you would like > > to test in qemu I recommend grabbing palmetto or romulus and doing > > that. So consider this dropping the generic qemu image and instead > > focusing on the aspeed one. > > +1 > > Many things are already broken on QEMU, including phosphor-ipmi-host. > It's not a useful platform to test with. > > > > > - add gsj. This gives us coverage of the nuvoton kernel and u-boot, > > as well as the nuvoton specific layers > > Agreed. I'd also like to switch to (or add) a Nuvoton system with a > host once such a machine is supported upstream. The gsj is only a > storage tray so it doesn't test IPMI bridges, power control, etc. > > > > > - add swift. This is an ast2500-based system that we're looking to > > use emmc flash with, and having testing for those images will be > > useful > > > > Cheers, > > > > Joel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: CI to stop testing meta-* layers not in tested machine 2019-06-21 0:38 ` Joel Stanley @ 2019-06-21 18:51 ` Andrew Geissler 0 siblings, 0 replies; 13+ messages in thread From: Andrew Geissler @ 2019-06-21 18:51 UTC (permalink / raw) To: Joel Stanley; +Cc: Benjamin Fair, OpenBMC Maillist, Patrick Venture On Thu, Jun 20, 2019 at 7:38 PM Joel Stanley <joel@jms.id.au> wrote: > > On Fri, 14 Jun 2019 at 15:55, Benjamin Fair <benjaminfair@google.com> wrote: > > > > > > Andrew tried to build the machine and ran into u-boot issues which is > > > still blocking the machine's addition to our CI. Patrick, are you able > > > to look into that? > > > > > > https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892 > > > > That issue will be resolved by switching to a 2019-based U-Boot branch: > > > > https://gerrit.openbmc-project.xyz/c/openbmc/meta-nuvoton/+/22556 > > This has been merged now. > > Andrew G, are we able to turn on the CI? > > I think we have consensus to drop qemu, and enable gsj. There were no > objections to enabling swift too. I removed qemu and added in gsj but we still need to work a few things out to get Swift going. I pinged you offline on that Joel. > > Cheers, > > Joel ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2019-06-21 18:51 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-03-14 13:38 CI to stop testing meta-* layers not in tested machine Andrew Geissler 2019-03-14 17:07 ` Patrick Venture 2019-05-09 19:36 ` Andrew Geissler 2019-05-10 1:10 ` Joel Stanley 2019-05-10 14:33 ` Patrick Venture 2019-05-10 18:45 ` Andrew Geissler 2019-06-14 4:54 ` Joel Stanley 2019-06-14 15:55 ` Benjamin Fair 2019-06-17 3:29 ` Andrew Jeffery 2019-06-19 0:45 ` Benjamin Fair 2019-06-19 1:06 ` Andrew Jeffery 2019-06-21 0:38 ` Joel Stanley 2019-06-21 18:51 ` Andrew Geissler
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.