All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: Network issue on 3.10 rcs, bisected
       [not found] <CACG0VPV3-x5Ncp+_hci8wVNYUeqAUZ0KFm0MAuHMntXzaL9Nqw@mail.gmail.com>
@ 2013-05-29 18:59 ` Pravin Shelar
  2013-05-29 19:53   ` Joao Correia
  0 siblings, 1 reply; 6+ messages in thread
From: Pravin Shelar @ 2013-05-29 18:59 UTC (permalink / raw)
  To: Joao Correia; +Cc: LKML, davem, netdev

On Wed, May 29, 2013 at 4:37 AM, Joao Correia
<joaomiguelcorreia@gmail.com> wrote:
> Hello list
>
> While trying the rc's for 3.10, i've stumbled upon a problem where
> networking does not work at all. Iptables will show packet counts going up,
> but nothing actually reaches the programs.
>
> I'm running fedora under hyper-v 3 (a windows 2012 host). Only tested ipv4
> traffic, and everything times out (ping, telnet to open ports) on both
> directions. The networking devices come up apparently ok - has static ip
> set, and dmesg shows no errors (although i don't have many debugging options
> enabled).
>
> I bisected this, and git blames commit
> ec5f061564238892005257c83565a0b58ec79295 (net: ill link between CSUM and SG
> features.). I can't revert it cleanly on current rc's.
>
Can you also send network features set on the device?
ethtool -k <dev>

Thanks,
Pravin.
> Please find the attached config used, as well as lshw output. Logs show no
> errors.
>
> I'm willing and able to test fixes.
>
> Thank you for your time,
> Joao Correia
> CIUBI
> Universidade da Beira Interior
> Portugal
>
>
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Network issue on 3.10 rcs, bisected
  2013-05-29 18:59 ` Network issue on 3.10 rcs, bisected Pravin Shelar
@ 2013-05-29 19:53   ` Joao Correia
  2013-05-30 16:48     ` Pravin Shelar
  0 siblings, 1 reply; 6+ messages in thread
From: Joao Correia @ 2013-05-29 19:53 UTC (permalink / raw)
  To: Pravin Shelar; +Cc: LKML, davem, netdev

On Wed, May 29, 2013 at 7:59 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>
> On Wed, May 29, 2013 at 4:37 AM, Joao Correia
> <joaomiguelcorreia@gmail.com> wrote:
> > Hello list
> >
> > While trying the rc's for 3.10, i've stumbled upon a problem where
> > networking does not work at all. Iptables will show packet counts going up,
> > but nothing actually reaches the programs.
> >
> > I'm running fedora under hyper-v 3 (a windows 2012 host). Only tested ipv4
> > traffic, and everything times out (ping, telnet to open ports) on both
> > directions. The networking devices come up apparently ok - has static ip
> > set, and dmesg shows no errors (although i don't have many debugging options
> > enabled).
> >
> > I bisected this, and git blames commit
> > ec5f061564238892005257c83565a0b58ec79295 (net: ill link between CSUM and SG
> > features.). I can't revert it cleanly on current rc's.
> >
> Can you also send network features set on the device?
> ethtool -k <dev>


As requested:
Features for eth0:
rx-checksumming: off [fixed]
tx-checksumming: off
    tx-checksum-ipv4: off [fixed]
    tx-checksum-ip-generic: off [fixed]
    tx-checksum-ipv6: off [fixed]
    tx-checksum-fcoe-crc: off [fixed]
    tx-checksum-sctp: off [fixed]
scatter-gather: on
    tx-scatter-gather: on
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
    tx-tcp-segmentation: off [fixed]
    tx-tcp-ecn-segmentation: off [fixed]
    tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
tx-vlan-ctag-hw-insert: on [fixed]
rx-vlan-ctag-hw-parse: off [fixed]
rx-vlan-ctag-filter: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]

The output is similar on a working (3.9) and a bad (3.10) kernel.
diff-ing both outputs shows:
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
as last on the broken kernels.

Thank you for your time,
Joao Correia
CIUBI
Universidade da Beira Interior
Portugal

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Network issue on 3.10 rcs, bisected
  2013-05-29 19:53   ` Joao Correia
@ 2013-05-30 16:48     ` Pravin Shelar
  2013-05-30 21:14       ` Joao Correia
  0 siblings, 1 reply; 6+ messages in thread
From: Pravin Shelar @ 2013-05-30 16:48 UTC (permalink / raw)
  To: Joao Correia; +Cc: LKML, davem, netdev

On Wed, May 29, 2013 at 12:53 PM, Joao Correia
<joaomiguelcorreia@gmail.com> wrote:
> On Wed, May 29, 2013 at 7:59 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>>
>> On Wed, May 29, 2013 at 4:37 AM, Joao Correia
>> <joaomiguelcorreia@gmail.com> wrote:
>> > Hello list
>> >
>> > While trying the rc's for 3.10, i've stumbled upon a problem where
>> > networking does not work at all. Iptables will show packet counts going up,
>> > but nothing actually reaches the programs.
>> >
>> > I'm running fedora under hyper-v 3 (a windows 2012 host). Only tested ipv4
>> > traffic, and everything times out (ping, telnet to open ports) on both
>> > directions. The networking devices come up apparently ok - has static ip
>> > set, and dmesg shows no errors (although i don't have many debugging options
>> > enabled).
>> >
>> > I bisected this, and git blames commit
>> > ec5f061564238892005257c83565a0b58ec79295 (net: ill link between CSUM and SG
>> > features.). I can't revert it cleanly on current rc's.
>> >
>> Can you also send network features set on the device?
>> ethtool -k <dev>
>
>
> As requested:
> Features for eth0:
> rx-checksumming: off [fixed]
> tx-checksumming: off
>     tx-checksum-ipv4: off [fixed]
>     tx-checksum-ip-generic: off [fixed]
>     tx-checksum-ipv6: off [fixed]
>     tx-checksum-fcoe-crc: off [fixed]
>     tx-checksum-sctp: off [fixed]
> scatter-gather: on
>     tx-scatter-gather: on
>     tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: off
>     tx-tcp-segmentation: off [fixed]
>     tx-tcp-ecn-segmentation: off [fixed]
>     tx-tcp6-segmentation: off [fixed]
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off
> tx-vlan-offload: on
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: off [fixed]
> tx-vlan-ctag-hw-insert: on [fixed]
> rx-vlan-ctag-hw-parse: off [fixed]
> rx-vlan-ctag-filter: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
>
> The output is similar on a working (3.9) and a bad (3.10) kernel.
> diff-ing both outputs shows:
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> as last on the broken kernels.
>

I could not reproduce it, I will try it on VM. Meanwhile can you turn
off feature "sg" and try same test ?

Thanks.

> Thank you for your time,
> Joao Correia
> CIUBI
> Universidade da Beira Interior
> Portugal

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Network issue on 3.10 rcs, bisected
  2013-05-30 16:48     ` Pravin Shelar
@ 2013-05-30 21:14       ` Joao Correia
  2013-05-30 23:13         ` Pravin Shelar
  0 siblings, 1 reply; 6+ messages in thread
From: Joao Correia @ 2013-05-30 21:14 UTC (permalink / raw)
  To: Pravin Shelar; +Cc: LKML, davem, netdev

On Thu, May 30, 2013 at 5:48 PM, Pravin Shelar <pshelar@nicira.com> wrote:
> On Wed, May 29, 2013 at 12:53 PM, Joao Correia
> <joaomiguelcorreia@gmail.com> wrote:
>> On Wed, May 29, 2013 at 7:59 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>>>
>>> On Wed, May 29, 2013 at 4:37 AM, Joao Correia
>>> <joaomiguelcorreia@gmail.com> wrote:
>>> > Hello list
>>> >
>>> > While trying the rc's for 3.10, i've stumbled upon a problem where
>>> > networking does not work at all. Iptables will show packet counts going up,
>>> > but nothing actually reaches the programs.
>>> >
>>> > I'm running fedora under hyper-v 3 (a windows 2012 host). Only tested ipv4
>>> > traffic, and everything times out (ping, telnet to open ports) on both
>>> > directions. The networking devices come up apparently ok - has static ip
>>> > set, and dmesg shows no errors (although i don't have many debugging options
>>> > enabled).
>>> >
>>> > I bisected this, and git blames commit
>>> > ec5f061564238892005257c83565a0b58ec79295 (net: ill link between CSUM and SG
>>> > features.). I can't revert it cleanly on current rc's.
>>> >
>>> Can you also send network features set on the device?
>>> ethtool -k <dev>
>>
>>
>> As requested:
>> Features for eth0:
>> rx-checksumming: off [fixed]
>> tx-checksumming: off
>>     tx-checksum-ipv4: off [fixed]
>>     tx-checksum-ip-generic: off [fixed]
>>     tx-checksum-ipv6: off [fixed]
>>     tx-checksum-fcoe-crc: off [fixed]
>>     tx-checksum-sctp: off [fixed]
>> scatter-gather: on
>>     tx-scatter-gather: on
>>     tx-scatter-gather-fraglist: off [fixed]
>> tcp-segmentation-offload: off
>>     tx-tcp-segmentation: off [fixed]
>>     tx-tcp-ecn-segmentation: off [fixed]
>>     tx-tcp6-segmentation: off [fixed]
>> udp-fragmentation-offload: off [fixed]
>> generic-segmentation-offload: on
>> generic-receive-offload: on
>> large-receive-offload: off [fixed]
>> rx-vlan-offload: off
>> tx-vlan-offload: on
>> ntuple-filters: off [fixed]
>> receive-hashing: off [fixed]
>> highdma: off [fixed]
>> tx-vlan-ctag-hw-insert: on [fixed]
>> rx-vlan-ctag-hw-parse: off [fixed]
>> rx-vlan-ctag-filter: off [fixed]
>> tx-vlan-stag-hw-insert: off [fixed]
>> rx-vlan-stag-hw-parse: off [fixed]
>> rx-vlan-stag-filter: off [fixed]
>> vlan-challenged: off [fixed]
>> tx-lockless: off [fixed]
>> netns-local: off [fixed]
>> tx-gso-robust: off [fixed]
>> tx-fcoe-segmentation: off [fixed]
>> tx-gre-segmentation: off [fixed]
>> tx-udp_tnl-segmentation: off [fixed]
>> fcoe-mtu: off [fixed]
>> tx-nocache-copy: off
>> loopback: off [fixed]
>> rx-fcs: off [fixed]
>> rx-all: off [fixed]
>>
>> The output is similar on a working (3.9) and a bad (3.10) kernel.
>> diff-ing both outputs shows:
>> tx-vlan-stag-hw-insert: off [fixed]
>> rx-vlan-stag-hw-parse: off [fixed]
>> rx-vlan-stag-filter: off [fixed]
>> as last on the broken kernels.
>>
>
> I could not reproduce it, I will try it on VM. Meanwhile can you turn
> off feature "sg" and try same test ?

Hello

Your hint was spot-on. With sg off, i can't reproduce the problem and
networking seems fine. It works fine either way on 3.9, so this is a
regression for 3.10.

Thank you very much for your assistance.
Joao Correia
CIUBI
Universidade da Beira Interior
Portugal

>
> Thanks.
>
>> Thank you for your time,
>> Joao Correia
>> CIUBI
>> Universidade da Beira Interior
>> Portugal

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Network issue on 3.10 rcs, bisected
  2013-05-30 21:14       ` Joao Correia
@ 2013-05-30 23:13         ` Pravin Shelar
  2013-05-31  8:26           ` Joao Correia
  0 siblings, 1 reply; 6+ messages in thread
From: Pravin Shelar @ 2013-05-30 23:13 UTC (permalink / raw)
  To: Joao Correia; +Cc: LKML, davem, netdev

On Thu, May 30, 2013 at 2:14 PM, Joao Correia
<joaomiguelcorreia@gmail.com> wrote:
> On Thu, May 30, 2013 at 5:48 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>> On Wed, May 29, 2013 at 12:53 PM, Joao Correia
>> <joaomiguelcorreia@gmail.com> wrote:
>>> On Wed, May 29, 2013 at 7:59 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>>>>
>>>> On Wed, May 29, 2013 at 4:37 AM, Joao Correia
>>>> <joaomiguelcorreia@gmail.com> wrote:
>>>> > Hello list
>>>> >
>>>> > While trying the rc's for 3.10, i've stumbled upon a problem where
>>>> > networking does not work at all. Iptables will show packet counts going up,
>>>> > but nothing actually reaches the programs.
>>>> >
>>>> > I'm running fedora under hyper-v 3 (a windows 2012 host). Only tested ipv4
>>>> > traffic, and everything times out (ping, telnet to open ports) on both
>>>> > directions. The networking devices come up apparently ok - has static ip
>>>> > set, and dmesg shows no errors (although i don't have many debugging options
>>>> > enabled).
>>>> >
>>>> > I bisected this, and git blames commit
>>>> > ec5f061564238892005257c83565a0b58ec79295 (net: ill link between CSUM and SG
>>>> > features.). I can't revert it cleanly on current rc's.
>>>> >
>>>> Can you also send network features set on the device?
>>>> ethtool -k <dev>
>>>
>>>
>>> As requested:
>>> Features for eth0:
>>> rx-checksumming: off [fixed]
>>> tx-checksumming: off
>>>     tx-checksum-ipv4: off [fixed]
>>>     tx-checksum-ip-generic: off [fixed]
>>>     tx-checksum-ipv6: off [fixed]
>>>     tx-checksum-fcoe-crc: off [fixed]
>>>     tx-checksum-sctp: off [fixed]
>>> scatter-gather: on
>>>     tx-scatter-gather: on
>>>     tx-scatter-gather-fraglist: off [fixed]
>>> tcp-segmentation-offload: off
>>>     tx-tcp-segmentation: off [fixed]
>>>     tx-tcp-ecn-segmentation: off [fixed]
>>>     tx-tcp6-segmentation: off [fixed]
>>> udp-fragmentation-offload: off [fixed]
>>> generic-segmentation-offload: on
>>> generic-receive-offload: on
>>> large-receive-offload: off [fixed]
>>> rx-vlan-offload: off
>>> tx-vlan-offload: on
>>> ntuple-filters: off [fixed]
>>> receive-hashing: off [fixed]
>>> highdma: off [fixed]
>>> tx-vlan-ctag-hw-insert: on [fixed]
>>> rx-vlan-ctag-hw-parse: off [fixed]
>>> rx-vlan-ctag-filter: off [fixed]
>>> tx-vlan-stag-hw-insert: off [fixed]
>>> rx-vlan-stag-hw-parse: off [fixed]
>>> rx-vlan-stag-filter: off [fixed]
>>> vlan-challenged: off [fixed]
>>> tx-lockless: off [fixed]
>>> netns-local: off [fixed]
>>> tx-gso-robust: off [fixed]
>>> tx-fcoe-segmentation: off [fixed]
>>> tx-gre-segmentation: off [fixed]
>>> tx-udp_tnl-segmentation: off [fixed]
>>> fcoe-mtu: off [fixed]
>>> tx-nocache-copy: off
>>> loopback: off [fixed]
>>> rx-fcs: off [fixed]
>>> rx-all: off [fixed]
>>>
>>> The output is similar on a working (3.9) and a bad (3.10) kernel.
>>> diff-ing both outputs shows:
>>> tx-vlan-stag-hw-insert: off [fixed]
>>> rx-vlan-stag-hw-parse: off [fixed]
>>> rx-vlan-stag-filter: off [fixed]
>>> as last on the broken kernels.
>>>
>>
>> I could not reproduce it, I will try it on VM. Meanwhile can you turn
>> off feature "sg" and try same test ?
>
> Hello
>
> Your hint was spot-on. With sg off, i can't reproduce the problem and
> networking seems fine. It works fine either way on 3.9, so this is a
> regression for 3.10.
>
Nice.
but still this does not look right. Can you tell me driver for the nic?
`ethtool -i <dev>`

Thanks,
Pravin.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Network issue on 3.10 rcs, bisected
  2013-05-30 23:13         ` Pravin Shelar
@ 2013-05-31  8:26           ` Joao Correia
  0 siblings, 0 replies; 6+ messages in thread
From: Joao Correia @ 2013-05-31  8:26 UTC (permalink / raw)
  To: Pravin Shelar; +Cc: LKML, davem, netdev

On Fri, May 31, 2013 at 12:13 AM, Pravin Shelar <pshelar@nicira.com> wrote:
> On Thu, May 30, 2013 at 2:14 PM, Joao Correia
> <joaomiguelcorreia@gmail.com> wrote:
>> On Thu, May 30, 2013 at 5:48 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>>> On Wed, May 29, 2013 at 12:53 PM, Joao Correia
>>> <joaomiguelcorreia@gmail.com> wrote:
>>>> On Wed, May 29, 2013 at 7:59 PM, Pravin Shelar <pshelar@nicira.com> wrote:
>>>>>
>>>>> On Wed, May 29, 2013 at 4:37 AM, Joao Correia
>>>>> <joaomiguelcorreia@gmail.com> wrote:
>>>>> > Hello list
>>>>> >
>>>>> > While trying the rc's for 3.10, i've stumbled upon a problem where
>>>>> > networking does not work at all. Iptables will show packet counts going up,
>>>>> > but nothing actually reaches the programs.
>>>>> >
>>>>> > I'm running fedora under hyper-v 3 (a windows 2012 host). Only tested ipv4
>>>>> > traffic, and everything times out (ping, telnet to open ports) on both
>>>>> > directions. The networking devices come up apparently ok - has static ip
>>>>> > set, and dmesg shows no errors (although i don't have many debugging options
>>>>> > enabled).
>>>>> >
>>>>> > I bisected this, and git blames commit
>>>>> > ec5f061564238892005257c83565a0b58ec79295 (net: ill link between CSUM and SG
>>>>> > features.). I can't revert it cleanly on current rc's.
>>>>> >
>>>>> Can you also send network features set on the device?
>>>>> ethtool -k <dev>
>>>>
>>>>
>>>> As requested:
>>>> Features for eth0:
>>>> rx-checksumming: off [fixed]
>>>> tx-checksumming: off
>>>>     tx-checksum-ipv4: off [fixed]
>>>>     tx-checksum-ip-generic: off [fixed]
>>>>     tx-checksum-ipv6: off [fixed]
>>>>     tx-checksum-fcoe-crc: off [fixed]
>>>>     tx-checksum-sctp: off [fixed]
>>>> scatter-gather: on
>>>>     tx-scatter-gather: on
>>>>     tx-scatter-gather-fraglist: off [fixed]
>>>> tcp-segmentation-offload: off
>>>>     tx-tcp-segmentation: off [fixed]
>>>>     tx-tcp-ecn-segmentation: off [fixed]
>>>>     tx-tcp6-segmentation: off [fixed]
>>>> udp-fragmentation-offload: off [fixed]
>>>> generic-segmentation-offload: on
>>>> generic-receive-offload: on
>>>> large-receive-offload: off [fixed]
>>>> rx-vlan-offload: off
>>>> tx-vlan-offload: on
>>>> ntuple-filters: off [fixed]
>>>> receive-hashing: off [fixed]
>>>> highdma: off [fixed]
>>>> tx-vlan-ctag-hw-insert: on [fixed]
>>>> rx-vlan-ctag-hw-parse: off [fixed]
>>>> rx-vlan-ctag-filter: off [fixed]
>>>> tx-vlan-stag-hw-insert: off [fixed]
>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>> rx-vlan-stag-filter: off [fixed]
>>>> vlan-challenged: off [fixed]
>>>> tx-lockless: off [fixed]
>>>> netns-local: off [fixed]
>>>> tx-gso-robust: off [fixed]
>>>> tx-fcoe-segmentation: off [fixed]
>>>> tx-gre-segmentation: off [fixed]
>>>> tx-udp_tnl-segmentation: off [fixed]
>>>> fcoe-mtu: off [fixed]
>>>> tx-nocache-copy: off
>>>> loopback: off [fixed]
>>>> rx-fcs: off [fixed]
>>>> rx-all: off [fixed]
>>>>
>>>> The output is similar on a working (3.9) and a bad (3.10) kernel.
>>>> diff-ing both outputs shows:
>>>> tx-vlan-stag-hw-insert: off [fixed]
>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>> rx-vlan-stag-filter: off [fixed]
>>>> as last on the broken kernels.
>>>>
>>>
>>> I could not reproduce it, I will try it on VM. Meanwhile can you turn
>>> off feature "sg" and try same test ?
>>
>> Hello
>>
>> Your hint was spot-on. With sg off, i can't reproduce the problem and
>> networking seems fine. It works fine either way on 3.9, so this is a
>> regression for 3.10.
>>
> Nice.
> but still this does not look right. Can you tell me driver for the nic?
> `ethtool -i <dev>`

ethtool -i eth0:
driver: hv_netvsc
version: 3.1
firmware-version: N/A
bus-info:
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

>
> Thanks,
> Pravin.

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2013-05-31  8:27 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <CACG0VPV3-x5Ncp+_hci8wVNYUeqAUZ0KFm0MAuHMntXzaL9Nqw@mail.gmail.com>
2013-05-29 18:59 ` Network issue on 3.10 rcs, bisected Pravin Shelar
2013-05-29 19:53   ` Joao Correia
2013-05-30 16:48     ` Pravin Shelar
2013-05-30 21:14       ` Joao Correia
2013-05-30 23:13         ` Pravin Shelar
2013-05-31  8:26           ` Joao Correia

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.