* dapltest-server segfault seen on recent OFED-1.5.4 daily build
@ 2011-11-18 9:01 Kumar Sanghvi
[not found] ` <20111118090155.GB17346-ZuiPNEE88OINxtijsoNbcrBI9BrxbZE7QQ4Iyu8u01E@public.gmane.org>
0 siblings, 1 reply; 6+ messages in thread
From: Kumar Sanghvi @ 2011-11-18 9:01 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Cc: arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w,
swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW,
divy-ut6Up61K2wZBDgjK7y7TUQ
Hi All,
I am trying to debug a segfault observed on dapltest-server with OFED-1.5.4.
I am using the daily-build OFED-1.5.4-20111116-0600 for this test.
The test setup involves 4 machines connected via switch.
1 machine acts as dapltest-server while rest 3 machines act as dapltest clients.
We are running several different kinds of RDMA read/write tests on dapl in continuous
loop using a script. The test runs fine for around 2 hours or so. And after that, the
dapltest-server segfaults with below stack trace:
-----------
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6e30710 (LWP 2397)]
dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
dapl/common/dapl_llist.c:272
272 dapl/common/dapl_llist.c: No such file or directory.
in dapl/common/dapl_llist.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.7.el6.x86_64
libgcc-4.4.4-13.el6.x86_64
(gdb) bt
#0 dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
dapl/common/dapl_llist.c:272
#1 0x00007ffff799fb09 in dapl_sp_remove_cr (sp_ptr=0x6368c0,
cr_ptr=0x7ffff0004be0) at dapl/common/dapl_sp_util.c:229
#2 0x00007ffff7998148 in dapli_connection_request (ib_cm_handle=<value
optimized out>, sp_ptr=0x6368c0, prd_ptr=<value optimized out>,
private_data_size=<value optimized out>, evd_ptr=0x633fb0) at
dapl/common/dapl_cr_callback.c:424
#3 0x00007ffff799851e in dapls_cr_callback (ib_cm_handle=0x7ffff0004880,
ib_cm_event=IB_CME_CONNECTION_REQUEST_PENDING,
private_data_ptr=0x0, private_data_size=0, context=0x6368c0) at
dapl/common/dapl_cr_callback.c:178
#4 0x00007ffff79a4c33 in dapli_cm_passive_cb () at dapl/openib_cma/cm.c:524
#5 dapli_cma_event_cb () at dapl/openib_cma/cm.c:1207
#6 0x00007ffff79a6657 in dapli_thread (arg=<value optimized out>) at
dapl/openib_cma/device.c:692
#7 0x00007ffff79971d1 in dapli_thread_init (thread_draft=0x630320) at
dapl/udapl/linux/dapl_osd.c:590
#8 0x0000003b156077e1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003b14ee153d in clone () from /lib64/libc.so.6
(gdb) p
The history is empty.
(gdb) info args
head = 0x636960
entry = 0x7ffff0004bf8
(gdb) p *head
$1 = (DAPL_LLIST_HEAD) 0x7ffff00107d8
(gdb) p *entry
$2 = {flink = 0x0, blink = 0x7ffff0003cf8, data = 0x7ffff0004be0, list_head =
0x0}
(gdb) info thread
950 Thread 0x7ffff7fef710 (LWP 3924) 0x0000003b1560b43c in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
949 Thread 0x7ffff5f76710 (LWP 3923) 0x0000003b1560eced in nanosleep () from
/lib64/libpthread.so.0
* 2 Thread 0x7ffff6e30710 (LWP 2397) dapl_llist_remove_entry (head=0x636960,
entry=0x7ffff0004bf8) at dapl/common/dapl_llist.c:272
1 Thread 0x7ffff7bb0700 (LWP 2394) 0x0000003b14ed7e33 in poll () from
/lib64/libc.so.6
(gdb) p head
$3 = (DAPL_LLIST_HEAD *) 0x636960
(gdb) p entry
$4 = (DAPL_LLIST_ENTRY *) 0x7ffff0004bf8
(gdb) p *entry
$5 = {flink = 0x0, blink = 0x7ffff0003cf8, data = 0x7ffff0004be0, list_head =
0x0}
(gdb) p *head
$6 = (DAPL_LLIST_HEAD) 0x7ffff00107d8
(gdb)
-----------
The problematic line in dapl source code is:
-------------
File dapl/common/dapl_llist.c#dapl_llist_remove_entry function:
....
dapl_os_assert(entry->list_head == head);
entry->list_head = NULL;
entry->flink->blink = entry->blink; <===== Problem line. flink is NULL
entry->blink->flink = entry->flink;
....
--------------
Now, it seems that some time back, a new release of dapl (dapl-2.0.34-1.src.rpm) was
introduced in OFED-1.5.4. So, I am just wondering if this is a regression in the new
release of dapl?
Or if anyone is aware of this issue and what could possibly lead to this
dapltest-server segfault then, it would be helpful if someone can shed some light.
Thanks,
Kumar.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build
[not found] ` <20111118090155.GB17346-ZuiPNEE88OINxtijsoNbcrBI9BrxbZE7QQ4Iyu8u01E@public.gmane.org>
@ 2011-11-18 23:48 ` Davis, Arlin R
[not found] ` <54347E5A035A054EAE9D05927FB467F916EA49A5-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
0 siblings, 1 reply; 6+ messages in thread
From: Davis, Arlin R @ 2011-11-18 23:48 UTC (permalink / raw)
To: Kumar Sanghvi, linux-rdma-u79uwXL29TY76Z2rM5mHXA
Cc: swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW, divy-ut6Up61K2wZBDgjK7y7TUQ
> #0 dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
> dapl/common/dapl_llist.c:272
> #1 0x00007ffff799fb09 in dapl_sp_remove_cr (sp_ptr=0x6368c0,
> cr_ptr=0x7ffff0004be0) at dapl/common/dapl_sp_util.c:229
> #2 0x00007ffff7998148 in dapli_connection_request (ib_cm_handle=<value
> optimized out>, sp_ptr=0x6368c0, prd_ptr=<value optimized out>,
> private_data_size=<value optimized out>, evd_ptr=0x633fb0) at
> dapl/common/dapl_cr_callback.c:424
>
> ...
>
> Now, it seems that some time back, a new release of dapl (dapl-2.0.34-
> 1.src.rpm) was
> introduced in OFED-1.5.4. So, I am just wondering if this is a
> regression in the new
> release of dapl?
> Or if anyone is aware of this issue and what could possibly lead to
> this
> dapltest-server segfault then, it would be helpful if someone can shed
> some light.
>
You should have seen a message like "WARNING: overflow event on EVD".
It appears that the default dapltest server allocates too small of a CR EVD for many client test configurations. When it hits the overflow queue case, the CR callback incorrectly frees the CR before it is removed from SP list. In your case, I am guessing that another CR came in on another thread and this memory was reallocated with flink ptr reinitialized.
Please try the following patches.
---------
Common: CR EVD overflow causes segfault.
The CR is freed up incorrectly before unlinking with SP.
Signed-off-by: Arlin Davis <arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
diff --git a/dapl/common/dapl_cr_callback.c b/dapl/common/dapl_cr_callback.c
index 3997b38..c58444b 100644
--- a/dapl/common/dapl_cr_callback.c
+++ b/dapl/common/dapl_cr_callback.c
@@ -414,7 +414,6 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
(DAT_CR_HANDLE) cr_ptr);
if (dat_status != DAT_SUCCESS) {
- dapls_cr_free(cr_ptr);
(void)dapls_ib_reject_connection(ib_cm_handle,
DAT_CONNECTION_EVENT_BROKEN,
0, NULL);
@@ -423,6 +422,7 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
dapl_os_lock(&sp_ptr->header.lock);
dapl_sp_remove_cr(sp_ptr, cr_ptr);
dapl_os_unlock(&sp_ptr->header.lock);
+ dapls_cr_free(cr_ptr);
return DAT_INSUFFICIENT_RESOURCES;
}
----------
dapltest: server CR EVD is too small for multi-client configurations.
Increase default size from 8 to 32.
Signed-off-by: Arlin Davis <arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
diff --git a/test/dapltest/test/dapl_server.c b/test/dapltest/test/dapl_server.c
index 443425c..92e0d21 100644
--- a/test/dapltest/test/dapl_server.c
+++ b/test/dapltest/test/dapl_server.c
@@ -34,7 +34,7 @@
#undef DFLT_QLEN
#endif
-#define DFLT_QLEN 8 /* default event queue length */
+#define DFLT_QLEN 32 /* default event queue length */
int send_control_data(DT_Tdep_Print_Head * phead,
unsigned char *buffp,
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build
[not found] ` <54347E5A035A054EAE9D05927FB467F916EA49A5-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
@ 2011-11-21 11:20 ` Kumar Sanghvi
[not found] ` <4ECA3402.8030203-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org>
0 siblings, 1 reply; 6+ messages in thread
From: Kumar Sanghvi @ 2011-11-21 11:20 UTC (permalink / raw)
To: Davis, Arlin R
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW,
divy-ut6Up61K2wZBDgjK7y7TUQ
Hi,
On 11/19/2011 05:18 AM, Davis, Arlin R wrote:
>
>> #0 dapl_llist_remove_entry (head=0x636960, entry=0x7ffff0004bf8) at
[...]
>>
>
> You should have seen a message like "WARNING: overflow event on EVD".
>
> It appears that the default dapltest server allocates too small of a CR EVD for many client test configurations. When it hits the overflow queue case, the CR callback incorrectly frees the CR before it is removed from SP list. In your case, I am guessing that another CR came in on another thread and this memory was reallocated with flink ptr reinitialized.
>
> Please try the following patches.
>
> ---------
> Common: CR EVD overflow causes segfault.
>
> The CR is freed up incorrectly before unlinking with SP.
>
> Signed-off-by: Arlin Davis<arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
>
> diff --git a/dapl/common/dapl_cr_callback.c b/dapl/common/dapl_cr_callback.c
> index 3997b38..c58444b 100644
> --- a/dapl/common/dapl_cr_callback.c
> +++ b/dapl/common/dapl_cr_callback.c
> @@ -414,7 +414,6 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
> (DAT_CR_HANDLE) cr_ptr);
>
> if (dat_status != DAT_SUCCESS) {
> - dapls_cr_free(cr_ptr);
> (void)dapls_ib_reject_connection(ib_cm_handle,
> DAT_CONNECTION_EVENT_BROKEN,
> 0, NULL);
> @@ -423,6 +422,7 @@ dapli_connection_request(IN dp_ib_cm_handle_t ib_cm_handle,
> dapl_os_lock(&sp_ptr->header.lock);
> dapl_sp_remove_cr(sp_ptr, cr_ptr);
> dapl_os_unlock(&sp_ptr->header.lock);
> + dapls_cr_free(cr_ptr);
> return DAT_INSUFFICIENT_RESOURCES;
> }
>
>
> ----------
> dapltest: server CR EVD is too small for multi-client configurations.
>
> Increase default size from 8 to 32.
>
> Signed-off-by: Arlin Davis<arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> diff --git a/test/dapltest/test/dapl_server.c b/test/dapltest/test/dapl_server.c
> index 443425c..92e0d21 100644
> --- a/test/dapltest/test/dapl_server.c
> +++ b/test/dapltest/test/dapl_server.c
> @@ -34,7 +34,7 @@
> #undef DFLT_QLEN
> #endif
>
> -#define DFLT_QLEN 8 /* default event queue length */
> +#define DFLT_QLEN 32 /* default event queue length */
>
> int send_control_data(DT_Tdep_Print_Head * phead,
> unsigned char *buffp,
>
>
Thank you for the two patches. I tried the two patches and now, I have
not seen a segfault till now on dapl-server at least.
However, after about 2 hours of test, some of dapl-client throws below
error on console:
----
Server Name: 3.4.5.1
Server Net Address: 3.4.5.1
DT_cs_Client: Starting Test ...
FAIL: 16 Server test connections did not report ready.
FAIL: 16 Server test connections did not report ready.
----
dapl-client is stalled at this stage, and needs to be manually killed by
Ctrl+C.
And below errors are seen on dapl-server console:
----
Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
Test[b368]: Warning: dat_ep_disconnect (abrupt) #2 error
DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE
DAT_INVALID_STATE_EVD_IN_USE
Test[b368]: Warning: dat_ep_disconnect (abrupt) #3 error
DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE
DAT_INVALID_STATE_EVD_IN_USE
...
----
No message is seen in dmesg on either dapl-server or dapl-client machine.
If I manually kill the dapl-client, and restart it then, test again
starts fine and runs for about 2 hours or so.
Thanks,
Kumar.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build
[not found] ` <4ECA3402.8030203-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org>
@ 2011-11-21 20:21 ` Davis, Arlin R
[not found] ` <54347E5A035A054EAE9D05927FB467F916EA4CE2-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
0 siblings, 1 reply; 6+ messages in thread
From: Davis, Arlin R @ 2011-11-21 20:21 UTC (permalink / raw)
To: Kumar Sanghvi
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW,
divy-ut6Up61K2wZBDgjK7y7TUQ
> Thank you for the two patches. I tried the two patches and now, I have
> not seen a segfault till now on dapl-server at least.
> However, after about 2 hours of test, some of dapl-client throws below
> error on console:
> ----
> Server Name: 3.4.5.1
> Server Net Address: 3.4.5.1
> DT_cs_Client: Starting Test ...
> FAIL: 16 Server test connections did not report ready.
> FAIL: 16 Server test connections did not report ready.
> ----
>
> dapl-client is stalled at this stage, and needs to be manually killed
> by
> Ctrl+C.
> And below errors are seen on dapl-server console:
> ----
> Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
> Test Error: Client_Mem_Info_Send-reaping DTO problem, status = FAILURE
> Test[b368]: Warning: dat_ep_disconnect (abrupt) #2 error
> DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
> Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE
> DAT_INVALID_STATE_EVD_IN_USE
> Test[b368]: Warning: dat_ep_disconnect (abrupt) #3 error
> DAT_INVALID_STATE DAT_INVALID_STATE_EP_UNCONNECTED
> Test[b368]: dat_evd_free (creq) error: DAT_INVALID_STATE
> DAT_INVALID_STATE_EVD_IN_USE
> ...
> ----
Can you send me your scripts so I can attempt to duplicate?
Did this test setup run successfully before OFED 1.5.4 RC3 release?
Thanks,
Arlin
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build
[not found] ` <54347E5A035A054EAE9D05927FB467F916EA4CE2-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
@ 2011-11-22 7:11 ` Kumar Sanghvi
[not found] ` <4ECB4B17.20407-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org>
0 siblings, 1 reply; 6+ messages in thread
From: Kumar Sanghvi @ 2011-11-22 7:11 UTC (permalink / raw)
To: Davis, Arlin R
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW,
divy-ut6Up61K2wZBDgjK7y7TUQ
[-- Attachment #1: Type: text/plain, Size: 393 bytes --]
Hi Arlin,
On 11/22/2011 01:51 AM, Davis, Arlin R wrote:
> Can you send me your scripts so I can attempt to duplicate?
Find attached the script with this mail.
> Did this test setup run successfully before OFED 1.5.4 RC3 release?
I am not sure when it stopped working.
However, we have been running this script on dapl for quite some time,
and it used to work fine earlier.
Thanks,
Kumar.
[-- Attachment #2: dapltest_stress_bug.txt --]
[-- Type: text/plain, Size: 2098 bytes --]
#!/bin/sh
host=$1
device=$2
while [ 1 ]
do
i=$(( i+1 ))
#====================================================================
#test1
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
client SR 256 \
server SR 256
#====================================================================
#test2
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
client SR 256 \
server RW 4096 \
server SR 256
#====================================================================
#test3
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
client SR 256 \
server RR 4096 \
server SR 256
#====================================================================
#test4
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 1 \
client SR 256 \
server RW 4096 \
server SR 256 \
client SR 256 \
server RW 4096 \
server SR 256 \
client SR 4096 \
server SR 256
#====================================================================
#test5
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 1 -w 8 \
client SR 256 \
server RW 4096 \
server SR 256 \
client SR 256 \
server RW 4096 \
server SR 256 \
client SR 4096 \
server SR 256
#====================================================================
#test6
#====================================================================
dapltest -T T -s ${host} -D ${device} -i 100000 -t 4 -w 8 \
client SR 256 \
server RW 4096 \
server SR 256 \
client SR 256 \
server RW 4096 \
server SR 256 \
client SR 4096 \
server SR 256
wait
echo "Iteration $i"
sleep 10
done
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] dapltest-server segfault seen on recent OFED-1.5.4 daily build
[not found] ` <4ECB4B17.20407-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org>
@ 2011-11-29 10:37 ` Kumar Sanghvi
0 siblings, 0 replies; 6+ messages in thread
From: Kumar Sanghvi @ 2011-11-29 10:37 UTC (permalink / raw)
To: Davis, Arlin R
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA,
swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW,
divy-ut6Up61K2wZBDgjK7y7TUQ
Hi Arlin,
Did you get a chance to look into this further?
Also, will you be releasing a new version of dapl with the two patches
provided earlier to OFED-1.5.4 ?
Thanks,
Kumar.
On 11/22/2011 12:41 PM, Kumar Sanghvi wrote:
> Hi Arlin,
>
> On 11/22/2011 01:51 AM, Davis, Arlin R wrote:
>> Can you send me your scripts so I can attempt to duplicate?
> Find attached the script with this mail.
>
>> Did this test setup run successfully before OFED 1.5.4 RC3 release?
> I am not sure when it stopped working.
> However, we have been running this script on dapl for quite some time,
> and it used to work fine earlier.
>
>
> Thanks,
> Kumar.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-11-29 10:37 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-11-18 9:01 dapltest-server segfault seen on recent OFED-1.5.4 daily build Kumar Sanghvi
[not found] ` <20111118090155.GB17346-ZuiPNEE88OINxtijsoNbcrBI9BrxbZE7QQ4Iyu8u01E@public.gmane.org>
2011-11-18 23:48 ` [PATCH] " Davis, Arlin R
[not found] ` <54347E5A035A054EAE9D05927FB467F916EA49A5-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
2011-11-21 11:20 ` Kumar Sanghvi
[not found] ` <4ECA3402.8030203-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org>
2011-11-21 20:21 ` Davis, Arlin R
[not found] ` <54347E5A035A054EAE9D05927FB467F916EA4CE2-P5GAC/sN6hmkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org>
2011-11-22 7:11 ` Kumar Sanghvi
[not found] ` <4ECB4B17.20407-ut6Up61K2wZBDgjK7y7TUQ@public.gmane.org>
2011-11-29 10:37 ` Kumar Sanghvi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.