* [linux-lvm] Hide volume group during startup [not found] <2049162317.298117.1334137791059.JavaMail.root@nelson.canoo.com> @ 2012-04-11 10:04 ` Erik Schwalbe 2012-04-12 11:11 ` Bryn M. Reeves 0 siblings, 1 reply; 13+ messages in thread From: Erik Schwalbe @ 2012-04-11 10:04 UTC (permalink / raw) To: linux-lvm [-- Attachment #1: Type: text/plain, Size: 824 bytes --] Hi, we have a problem with hiding/deactivating a stacked volume group during startup. System is a Ubuntu 10.4.4 LTS (NAS). lvm2: 2.02.54-1ubuntu4.1ppa5 lvs: backup-all data -wi-ao 3.00t fileserver-forBackup data -wi-ao 50.00g .... kvm_disk0 data -wi-ao 400.00g vm-203-disk-1 kvm_disk0 -wi-ao- 28.01g .... kvm_disk0 is used as device for iscsitarget. The iscsiinitiator created a new volume group (kvm_disk0) on the blockdevice and new virtual disks (vm-203-disk-1) . Our problem is, that the iscsitarget host found not only the VG "data" during system startup but also the other VG (kvm_disk0) and activate it. So DRBD can't get exclusive access to the LV kvm_disk0. We try to filter in lvm.conf, but after startup, the VG kvm_disk0 is allways activated. Thank you for help. Regards, Erik [-- Attachment #2: Type: text/html, Size: 2871 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-11 10:04 ` [linux-lvm] Hide volume group during startup Erik Schwalbe @ 2012-04-12 11:11 ` Bryn M. Reeves 2012-04-12 11:41 ` Erik Schwalbe 0 siblings, 1 reply; 13+ messages in thread From: Bryn M. Reeves @ 2012-04-12 11:11 UTC (permalink / raw) To: LVM general discussion and development; +Cc: Erik Schwalbe On 04/11/2012 11:04 AM, Erik Schwalbe wrote: > The iscsiinitiator created a new volume group (kvm_disk0) on the > blockdevice and new virtual disks (vm-203-disk-1). > > Our problem is, that the iscsitarget host found not only the VG "data" > during system startup but also the other VG (kvm_disk0) and activate it. > So DRBD can't get exclusive access to the LV kvm_disk0. > > We try to filter in lvm.conf, but after startup, the VG kvm_disk0 is > allways activated. Please post the filter you are using as well as the full path name of the disk you are attempting to filter (a complete recursive listing of /dev as generated by lvmdump or run by hand would be useful. You can use a service like pastebin for this - please don't send large attachments to the list). There are other ways to control activation if for some reason filtering will not work (e.g. the device name is not stable). Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 11:11 ` Bryn M. Reeves @ 2012-04-12 11:41 ` Erik Schwalbe 2012-04-12 11:52 ` Bryn M. Reeves 2012-04-12 12:01 ` Lars Ellenberg 0 siblings, 2 replies; 13+ messages in thread From: Erik Schwalbe @ 2012-04-12 11:41 UTC (permalink / raw) To: LVM general discussion and development Hi, filter was: filter = [ "r|/dev/data/kvm_disk0|" ] path: /dev/data/kvm_disk0 -> ../mapper/data-kvm_disk0 The device name is stable, but what other way of control activation are possible?? Regards, Erik ----- Original Message ----- From: "Bryn M. Reeves" <bmr@redhat.com> To: "LVM general discussion and development" <linux-lvm@redhat.com> Cc: "Erik Schwalbe" <erik.schwalbe@canoo.com> Sent: Thursday, 12 April, 2012 1:11:44 PM Subject: Re: [linux-lvm] Hide volume group during startup On 04/11/2012 11:04 AM, Erik Schwalbe wrote: > The iscsiinitiator created a new volume group (kvm_disk0) on the > blockdevice and new virtual disks (vm-203-disk-1). > > Our problem is, that the iscsitarget host found not only the VG "data" > during system startup but also the other VG (kvm_disk0) and activate it. > So DRBD can't get exclusive access to the LV kvm_disk0. > > We try to filter in lvm.conf, but after startup, the VG kvm_disk0 is > allways activated. Please post the filter you are using as well as the full path name of the disk you are attempting to filter (a complete recursive listing of /dev as generated by lvmdump or run by hand would be useful. You can use a service like pastebin for this - please don't send large attachments to the list). There are other ways to control activation if for some reason filtering will not work (e.g. the device name is not stable). Regards, Bryn. _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 11:41 ` Erik Schwalbe @ 2012-04-12 11:52 ` Bryn M. Reeves 2012-04-12 12:14 ` Erik Schwalbe 2012-04-12 12:01 ` Lars Ellenberg 1 sibling, 1 reply; 13+ messages in thread From: Bryn M. Reeves @ 2012-04-12 11:52 UTC (permalink / raw) To: LVM general discussion and development; +Cc: Erik Schwalbe On 04/12/2012 12:41 PM, Erik Schwalbe wrote: > Hi, > > filter was: > > filter = [ "r|/dev/data/kvm_disk0|" ] > > path: > /dev/data/kvm_disk0 -> ../mapper/data-kvm_disk0 To filter a device-mapper device you also have to filter out the /dev/dm-N device nodes that udev manages. There's also a symlink from /dev/$VG_NAME/$LV_NAME to the dm-N node (although I think that should be handled by filtering /dev/dm-N). I'd be using something like: [ "r|/dev/dm.*|", "r|/dev/$VG_NAME|", "a|.*|"] Alternately (and I prefer this for many configurations) you can accept ("a|..|") the device you need for the data VG and reject all other devices ("r|.*|"). > The device name is stable, but what other way of control activation are possible?? Tags but since you're not currently filtering dm-N nodes I would try that first. Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 11:52 ` Bryn M. Reeves @ 2012-04-12 12:14 ` Erik Schwalbe 2012-04-12 12:26 ` Bryn M. Reeves 0 siblings, 1 reply; 13+ messages in thread From: Erik Schwalbe @ 2012-04-12 12:14 UTC (permalink / raw) To: LVM general discussion and development Hi, there is only one dm-device in /dev: ... drwxr-xr-x 5 root root 100 2012-04-05 12:50 disk brw-rw---- 1 root disk 251, 18 2012-04-05 12:51 dm-18 drwxr-xr-x 4 root root 80 2012-04-05 12:51 drbd ... /dev/data/kvm_disk0 is a blockdevice for drbd, so there are many drbd files in /dev ... brw-rw---- 1 root disk 147, 20 2012-04-08 11:48 drbd20 ... drbd20 is the drbd resource using kvm_disk0 as device. The symlink in /dev/data/ is: lrwxrwxrwx 1 root root 24 2012-04-05 12:51 kvm_disk0 -> ../mapper/data-kvm_disk0 Should I be using this: filter = [ "r|/dev/mapper/data-kvm_disk0|", "r|/dev/kvm_disk0|", "a|.*|"] I know, it is a little bit confusing. The LV in VG data is called kvm_disk0 and the VG initiated on this device is also called kvm_disk0. Sorry, but thank you for help. Regards, Erik ----- Original Message ----- From: "Bryn M. Reeves" <bmr@redhat.com> To: "LVM general discussion and development" <linux-lvm@redhat.com> Cc: "Erik Schwalbe" <erik.schwalbe@canoo.com> Sent: Thursday, 12 April, 2012 1:52:20 PM Subject: Re: [linux-lvm] Hide volume group during startup On 04/12/2012 12:41 PM, Erik Schwalbe wrote: > Hi, > > filter was: > > filter = [ "r|/dev/data/kvm_disk0|" ] > > path: > /dev/data/kvm_disk0 -> ../mapper/data-kvm_disk0 To filter a device-mapper device you also have to filter out the /dev/dm-N device nodes that udev manages. There's also a symlink from /dev/$VG_NAME/$LV_NAME to the dm-N node (although I think that should be handled by filtering /dev/dm-N). I'd be using something like: [ "r|/dev/dm.*|", "r|/dev/$VG_NAME|", "a|.*|"] Alternately (and I prefer this for many configurations) you can accept ("a|..|") the device you need for the data VG and reject all other devices ("r|.*|"). > The device name is stable, but what other way of control activation are possible?? Tags but since you're not currently filtering dm-N nodes I would try that first. Regards, Bryn. _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 12:14 ` Erik Schwalbe @ 2012-04-12 12:26 ` Bryn M. Reeves 2012-04-12 13:31 ` Erik Schwalbe 0 siblings, 1 reply; 13+ messages in thread From: Bryn M. Reeves @ 2012-04-12 12:26 UTC (permalink / raw) To: LVM general discussion and development; +Cc: Erik Schwalbe On 04/12/2012 01:14 PM, Erik Schwalbe wrote: > The symlink in /dev/data/ is: > > lrwxrwxrwx 1 root root 24 2012-04-05 12:51 kvm_disk0 -> ../mapper/data-kvm_disk0 OK, you're using an older LVM2/device-mapper that does not use udev to manage the device nodes and symlinks in /dev. > Should I be using this: > > filter = [ "r|/dev/mapper/data-kvm_disk0|", "r|/dev/kvm_disk0|", "a|.*|"] Yes. Or use the alternate style I showed and only accept the PVs belonging to the data VG. As Lars said you may also need to rebuild your initramfs although if the data VG doesn't contain your root file system, swap or other resources normally set up in the initramfs this shouldn't be needed. Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 12:26 ` Bryn M. Reeves @ 2012-04-12 13:31 ` Erik Schwalbe 2012-04-12 13:39 ` Bryn M. Reeves 0 siblings, 1 reply; 13+ messages in thread From: Erik Schwalbe @ 2012-04-12 13:31 UTC (permalink / raw) To: Bryn M. Reeves; +Cc: LVM general discussion and development Hi, sorry but it did not work: I deactivate the VG kvm_disk0 with vgchange -an kvm_disk0, after that I set the filter and do a update-initramfs -u -k all But after reboot the VG is always activate. ... kvm_disk0 data -wi-ao 400.00g vm-203-disk-1 kvm_disk0 -wi-a- 28.01g vm-203-disk-2 kvm_disk0 -wi-a- 80.00g ... Regards, Erik ----- Original Message ----- From: "Bryn M. Reeves" <bmr@redhat.com> To: "LVM general discussion and development" <linux-lvm@redhat.com> Cc: "Erik Schwalbe" <erik.schwalbe@canoo.com> Sent: Thursday, 12 April, 2012 2:26:34 PM Subject: Re: [linux-lvm] Hide volume group during startup On 04/12/2012 01:14 PM, Erik Schwalbe wrote: > The symlink in /dev/data/ is: > > lrwxrwxrwx 1 root root 24 2012-04-05 12:51 kvm_disk0 -> ../mapper/data-kvm_disk0 OK, you're using an older LVM2/device-mapper that does not use udev to manage the device nodes and symlinks in /dev. > Should I be using this: > > filter = [ "r|/dev/mapper/data-kvm_disk0|", "r|/dev/kvm_disk0|", "a|.*|"] Yes. Or use the alternate style I showed and only accept the PVs belonging to the data VG. As Lars said you may also need to rebuild your initramfs although if the data VG doesn't contain your root file system, swap or other resources normally set up in the initramfs this shouldn't be needed. Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 13:31 ` Erik Schwalbe @ 2012-04-12 13:39 ` Bryn M. Reeves 2012-04-12 14:14 ` Erik Schwalbe 0 siblings, 1 reply; 13+ messages in thread From: Bryn M. Reeves @ 2012-04-12 13:39 UTC (permalink / raw) To: Erik Schwalbe; +Cc: LVM general discussion and development On 04/12/2012 02:31 PM, Erik Schwalbe wrote: > Hi, > > sorry but it did not work: > > I deactivate the VG kvm_disk0 with vgchange -an kvm_disk0, after that I set the filter Test it at this point. There is no need to reboot. If pvs/vgchange/vgdisplay etc. still display the nested VG and its PV your filters are wrong or insufficient. Examine the -vvv output from the tools to see the filter decisions that are being made. Since you haven't mentioned any of the versions you are using you may also want to remove the LVM2 cache after editing the filter (current versions clear it automatically but older releases would still show filtered devices if they exist in the cache. > and do a update-initramfs -u -k all You need to know whether this is the correct command for your distribution and that it's affecting whatever kernel/initramfs combination the machine boots by default (and also that it actually works and creates a new initramfs). But for now I would ignore boot problems and focus on the filter. Don't try to solve all the problems at once as that just makes it more confusing. When you have confirmed that your filter settings correctly exclude the nested VG you can move on to making sure those settings are applied at boot. Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 13:39 ` Bryn M. Reeves @ 2012-04-12 14:14 ` Erik Schwalbe 2012-04-12 14:30 ` Bryn M. Reeves 0 siblings, 1 reply; 13+ messages in thread From: Erik Schwalbe @ 2012-04-12 14:14 UTC (permalink / raw) To: Bryn M. Reeves; +Cc: LVM general discussion and development Ok, I found a solution: I deactivate the VG kvm_disk0 with vgchange -an kvm_disk0, after that I set the filter: filter = [ "r|/dev/mapper/data-kvm_disk0|", "r|/dev/kvm_disk0|"] without "a|.*|" !!!! and do a update-initramfs -u -k all Perfect :-) It seems that the allow filter overrules the remove option. Is my filter correct?? ps: lvm version: 2.02.54-1ubuntu4.1ppa5 Regards, Erik ----- Original Message ----- From: "Bryn M. Reeves" <bmr@redhat.com> To: "Erik Schwalbe" <erik.schwalbe@canoo.com> Cc: "LVM general discussion and development" <linux-lvm@redhat.com> Sent: Thursday, 12 April, 2012 3:39:55 PM Subject: Re: [linux-lvm] Hide volume group during startup On 04/12/2012 02:31 PM, Erik Schwalbe wrote: > Hi, > > sorry but it did not work: > > I deactivate the VG kvm_disk0 with vgchange -an kvm_disk0, after that I set the filter Test it at this point. There is no need to reboot. If pvs/vgchange/vgdisplay etc. still display the nested VG and its PV your filters are wrong or insufficient. Examine the -vvv output from the tools to see the filter decisions that are being made. Since you haven't mentioned any of the versions you are using you may also want to remove the LVM2 cache after editing the filter (current versions clear it automatically but older releases would still show filtered devices if they exist in the cache. > and do a update-initramfs -u -k all You need to know whether this is the correct command for your distribution and that it's affecting whatever kernel/initramfs combination the machine boots by default (and also that it actually works and creates a new initramfs). But for now I would ignore boot problems and focus on the filter. Don't try to solve all the problems at once as that just makes it more confusing. When you have confirmed that your filter settings correctly exclude the nested VG you can move on to making sure those settings are applied at boot. Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 14:14 ` Erik Schwalbe @ 2012-04-12 14:30 ` Bryn M. Reeves 2012-04-12 14:50 ` Erik Schwalbe 0 siblings, 1 reply; 13+ messages in thread From: Bryn M. Reeves @ 2012-04-12 14:30 UTC (permalink / raw) To: Erik Schwalbe; +Cc: LVM general discussion and development On 04/12/2012 03:14 PM, Erik Schwalbe wrote: > Ok, I found a solution: > > I deactivate the VG kvm_disk0 with vgchange -an kvm_disk0, after that I set the filter: > > filter = [ "r|/dev/mapper/data-kvm_disk0|", "r|/dev/kvm_disk0|"] > > without "a|.*|" !!!! Shouldn't make any difference. The filter line has an implicit "a|.*|" at the end. From the lvm.conf man page: "Devices that don't match any patterns are accepted." > Is my filter correct?? If it works for you then that's the main thing. Just be aware that it will need adjustment if you add any more LVs that need to be treated this way (that's why I prefer to accept the PVs I need and reject everything else with a final "r|.*|"). Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 14:30 ` Bryn M. Reeves @ 2012-04-12 14:50 ` Erik Schwalbe 0 siblings, 0 replies; 13+ messages in thread From: Erik Schwalbe @ 2012-04-12 14:50 UTC (permalink / raw) To: Bryn M. Reeves; +Cc: LVM general discussion and development Thank you very much for your help. Regards, Erik ----- Original Message ----- From: "Bryn M. Reeves" <bmr@redhat.com> To: "Erik Schwalbe" <erik.schwalbe@canoo.com> Cc: "LVM general discussion and development" <linux-lvm@redhat.com> Sent: Thursday, 12 April, 2012 4:30:16 PM Subject: Re: [linux-lvm] Hide volume group during startup On 04/12/2012 03:14 PM, Erik Schwalbe wrote: > Ok, I found a solution: > > I deactivate the VG kvm_disk0 with vgchange -an kvm_disk0, after that I set the filter: > > filter = [ "r|/dev/mapper/data-kvm_disk0|", "r|/dev/kvm_disk0|"] > > without "a|.*|" !!!! Shouldn't make any difference. The filter line has an implicit "a|.*|" at the end. From the lvm.conf man page: "Devices that don't match any patterns are accepted." > Is my filter correct?? If it works for you then that's the main thing. Just be aware that it will need adjustment if you add any more LVs that need to be treated this way (that's why I prefer to accept the PVs I need and reject everything else with a final "r|.*|"). Regards, Bryn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 11:41 ` Erik Schwalbe 2012-04-12 11:52 ` Bryn M. Reeves @ 2012-04-12 12:01 ` Lars Ellenberg 2012-04-12 12:33 ` Erik Schwalbe 1 sibling, 1 reply; 13+ messages in thread From: Lars Ellenberg @ 2012-04-12 12:01 UTC (permalink / raw) To: linux-lvm On Thu, Apr 12, 2012 at 01:41:31PM +0200, Erik Schwalbe wrote: > Hi, > > filter was: > > filter = [ "r|/dev/data/kvm_disk0|" ] > > path: > /dev/data/kvm_disk0 -> ../mapper/data-kvm_disk0 > > The device name is stable, but what other way of control activation are possible?? Forgot to update the initramfs/initrd, and the VGs are activated from there already, even before pivoting to the real root? Run update-initramfs or mkinitrd /boot/initramfs-`uname -r`.img `uname -r` or something like that. > > Regards, > Erik > > ----- Original Message ----- > From: "Bryn M. Reeves" <bmr@redhat.com> > To: "LVM general discussion and development" <linux-lvm@redhat.com> > Cc: "Erik Schwalbe" <erik.schwalbe@canoo.com> > Sent: Thursday, 12 April, 2012 1:11:44 PM > Subject: Re: [linux-lvm] Hide volume group during startup > > On 04/11/2012 11:04 AM, Erik Schwalbe wrote: > > The iscsiinitiator created a new volume group (kvm_disk0) on the > > blockdevice and new virtual disks (vm-203-disk-1). > > > > Our problem is, that the iscsitarget host found not only the VG "data" > > during system startup but also the other VG (kvm_disk0) and activate it. > > So DRBD can't get exclusive access to the LV kvm_disk0. > > > > We try to filter in lvm.conf, but after startup, the VG kvm_disk0 is > > allways activated. > > Please post the filter you are using as well as the full path name of > the disk you are attempting to filter (a complete recursive listing of > /dev as generated by lvmdump or run by hand would be useful. You can use > a service like pastebin for this - please don't send large attachments > to the list). > > There are other ways to control activation if for some reason filtering > will not work (e.g. the device name is not stable). > > Regards, > Bryn. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Hide volume group during startup 2012-04-12 12:01 ` Lars Ellenberg @ 2012-04-12 12:33 ` Erik Schwalbe 0 siblings, 0 replies; 13+ messages in thread From: Erik Schwalbe @ 2012-04-12 12:33 UTC (permalink / raw) To: LVM general discussion and development Ok, I will try a: update-initramfs -u -k all Hope this helps. Thank you for the hint. Regards, Erik ----- Original Message ----- From: "Lars Ellenberg" <lars.ellenberg@linbit.com> To: linux-lvm@redhat.com Sent: Thursday, 12 April, 2012 2:01:39 PM Subject: Re: [linux-lvm] Hide volume group during startup On Thu, Apr 12, 2012 at 01:41:31PM +0200, Erik Schwalbe wrote: > Hi, > > filter was: > > filter = [ "r|/dev/data/kvm_disk0|" ] > > path: > /dev/data/kvm_disk0 -> ../mapper/data-kvm_disk0 > > The device name is stable, but what other way of control activation are possible?? Forgot to update the initramfs/initrd, and the VGs are activated from there already, even before pivoting to the real root? Run update-initramfs or mkinitrd /boot/initramfs-`uname -r`.img `uname -r` or something like that. > > Regards, > Erik > > ----- Original Message ----- > From: "Bryn M. Reeves" <bmr@redhat.com> > To: "LVM general discussion and development" <linux-lvm@redhat.com> > Cc: "Erik Schwalbe" <erik.schwalbe@canoo.com> > Sent: Thursday, 12 April, 2012 1:11:44 PM > Subject: Re: [linux-lvm] Hide volume group during startup > > On 04/11/2012 11:04 AM, Erik Schwalbe wrote: > > The iscsiinitiator created a new volume group (kvm_disk0) on the > > blockdevice and new virtual disks (vm-203-disk-1). > > > > Our problem is, that the iscsitarget host found not only the VG "data" > > during system startup but also the other VG (kvm_disk0) and activate it. > > So DRBD can't get exclusive access to the LV kvm_disk0. > > > > We try to filter in lvm.conf, but after startup, the VG kvm_disk0 is > > allways activated. > > Please post the filter you are using as well as the full path name of > the disk you are attempting to filter (a complete recursive listing of > /dev as generated by lvmdump or run by hand would be useful. You can use > a service like pastebin for this - please don't send large attachments > to the list). > > There are other ways to control activation if for some reason filtering > will not work (e.g. the device name is not stable). > > Regards, > Bryn. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2012-04-12 14:50 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <2049162317.298117.1334137791059.JavaMail.root@nelson.canoo.com> 2012-04-11 10:04 ` [linux-lvm] Hide volume group during startup Erik Schwalbe 2012-04-12 11:11 ` Bryn M. Reeves 2012-04-12 11:41 ` Erik Schwalbe 2012-04-12 11:52 ` Bryn M. Reeves 2012-04-12 12:14 ` Erik Schwalbe 2012-04-12 12:26 ` Bryn M. Reeves 2012-04-12 13:31 ` Erik Schwalbe 2012-04-12 13:39 ` Bryn M. Reeves 2012-04-12 14:14 ` Erik Schwalbe 2012-04-12 14:30 ` Bryn M. Reeves 2012-04-12 14:50 ` Erik Schwalbe 2012-04-12 12:01 ` Lars Ellenberg 2012-04-12 12:33 ` Erik Schwalbe
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.