bonding with 3c59x driver

bonding with 3c59x driver

[Jean Delvare]

Hi Steffen, all,

I am trying to do network bonding on top of 3com 3C905C ethernet 
adapters. I am facing the problem that network cable removal isn't 
detected. It is detected when using Realtek hardware with driver 8139too 
or VIA hardware with driver via-rhine, but no luck with driver 3c59x. 
I'm using bonding option miimon=100 for link detection.

Does anyone know if this is a hardware limitation, or a missing feature 
of the 3c59x driver, or a bug in that driver?

Thanks,
-- 
Jean Delvare
Suse L3

Re: bonding with 3c59x driver

[Jay Vosburgh]

Jean Delvare <jdelvare@suse.de> wrote:
>I am trying to do network bonding on top of 3com 3C905C ethernet 
>adapters. I am facing the problem that network cable removal isn't 
>detected. It is detected when using Realtek hardware with driver 8139too 
>or VIA hardware with driver via-rhine, but no luck with driver 3c59x. 
>I'm using bonding option miimon=100 for link detection.
>
>Does anyone know if this is a hardware limitation, or a missing feature 
>of the 3c59x driver, or a bug in that driver?

	It just so happens that I was looking at 3c59x earlier today for
something else.  It appears to run its internal link state detection on
a 5 second timer, after which (in theory) bonding's miimon would notice
(provided that the driver notices and calls netif_carrier_off).

	If that's not working, did you try use_carrier=0 to probe the
mii register state directly?

	I have used 3c59x with bonding in the distant past, but I don't
recall exactly what tweaks were necessary; it may have required
use_carrier=0 to work at all.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: bonding with 3c59x driver

[Jean Delvare]

Hi Jay,

Thanks for the fast answer.

On Monday 13 February 2012 11:13:11 pm Jay Vosburgh wrote:
> Jean Delvare <jdelvare@suse.de> wrote:
> >I am trying to do network bonding on top of 3com 3C905C ethernet
> >adapters. I am facing the problem that network cable removal isn't
> >detected. It is detected when using Realtek hardware with driver
> > 8139too or VIA hardware with driver via-rhine, but no luck with
> > driver 3c59x. I'm using bonding option miimon=100 for link
> > detection.
> >
> >Does anyone know if this is a hardware limitation, or a missing
> > feature of the 3c59x driver, or a bug in that driver?
> 
> 	It just so happens that I was looking at 3c59x earlier today for
> something else.  It appears to run its internal link state detection
>  on a 5 second timer, after which (in theory) bonding's miimon would
>  notice (provided that the driver notices and calls
>  netif_carrier_off).

Thanks for the pointer. There is indeed a 5 second timer to detect 
transitions from link down to link up. However the timer period to 
detect transitions from link up to link down is 1 minute. Knowing that, 
I tested again and indeed the bonding module end up detecting the link 
going down, after up to 1 minute. That's not exactly convenient for 
bonding.

I admit I don't quite get the rationale for having different timer 
periods for detecting the link going up or down. Steffen, this was done 
by you in:

commit b4ff6450f5336c492d1e2f184d3b8186e0716b7a
Author: Steffen Klassert <klassert@mathematik.tu-chemnitz.de>
Date:   Sun Mar 26 01:37:40 2006 -0800

    [PATCH] 3c59x: decrease polling interval

No rationale explained in the commit message. Can you explain? I find 
the asymmetry confusing, I'd rather use 5 seconds (if not less) for both 
cases.

> 	If that's not working, did you try use_carrier=0 to probe the
> mii register state directly?

I tried use_carrier=0 but it makes no difference.

> 	I have used 3c59x with bonding in the distant past, but I don't
> recall exactly what tweaks were necessary; it may have required
> use_carrier=0 to work at all.

Well I am still curious why the 3c59x driver has such slow polling when 
other drivers I have been testing are able to report almost instantly 
when I remove a network cable. Could it be that other network chips 
generate an interrupt on cable removal and the 3com chips do not?

-- 
Jean Delvare
Suse L3

Re: bonding with 3c59x driver

[Steffen Klassert]

On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:
> Hi Jay,
> 
> Thanks for the fast answer.
> 
> On Monday 13 February 2012 11:13:11 pm Jay Vosburgh wrote:
> > Jean Delvare <jdelvare@suse.de> wrote:
> > >I am trying to do network bonding on top of 3com 3C905C ethernet
> > >adapters. I am facing the problem that network cable removal isn't
> > >detected. It is detected when using Realtek hardware with driver
> > > 8139too or VIA hardware with driver via-rhine, but no luck with
> > > driver 3c59x. I'm using bonding option miimon=100 for link
> > > detection.
> > >
> > >Does anyone know if this is a hardware limitation, or a missing
> > > feature of the 3c59x driver, or a bug in that driver?
> > 
> > 	It just so happens that I was looking at 3c59x earlier today for
> > something else.  It appears to run its internal link state detection
> >  on a 5 second timer, after which (in theory) bonding's miimon would
> >  notice (provided that the driver notices and calls
> >  netif_carrier_off).
> 
> Thanks for the pointer. There is indeed a 5 second timer to detect 
> transitions from link down to link up. However the timer period to 
> detect transitions from link up to link down is 1 minute. Knowing that, 
> I tested again and indeed the bonding module end up detecting the link 
> going down, after up to 1 minute. That's not exactly convenient for 
> bonding.
> 
> I admit I don't quite get the rationale for having different timer 
> periods for detecting the link going up or down. Steffen, this was done 
> by you in:
> 
> commit b4ff6450f5336c492d1e2f184d3b8186e0716b7a
> Author: Steffen Klassert <klassert@mathematik.tu-chemnitz.de>
> Date:   Sun Mar 26 01:37:40 2006 -0800
> 
>     [PATCH] 3c59x: decrease polling interval
> 
> No rationale explained in the commit message. Can you explain? I find 
> the asymmetry confusing, I'd rather use 5 seconds (if not less) for both 
> cases.


This patch is based on a discussion with Andrew Morton some years ago.
The 3c59x does polling for external environment changes which is quite
expensive. Firing a timer that disables the interrupts on a running
interface every 5 seconds is just not reasonable for checking for external
environment changes. So we decided to let it depend on the link status,
5 seconds if the link is down and 60 seconds if the link is up.

RE: bonding with 3c59x driver

[David Laight]

 
> Well I am still curious why the 3c59x driver has such slow 
> polling when 
> other drivers I have been testing are able to report almost instantly 
> when I remove a network cable. Could it be that other network chips 
> generate an interrupt on cable removal and the 3com chips do not?

Not sure about those specific cards, but detecting the
link state might require software bit-banging of an MII
(or similar) interface.

Since this needs delays between all the edges it can
end up spinning the cpu for a noticable time, especially
if the resolution (and minimim delay) of the delay()
function used is significantly longer than the delays
actually required - which might only be a few 100ns.

Using a cpu spin for the delays is also slightly
problematical because the writes might get 'posted'
and delayed further - so the writes have to be forced
out to the device itself (eg by a readback) prior to
the delay() being requested.

If the driver is polling the link status it might
be valid to do one bit-bang transition on each timer
interrupt. This would separate the edges without
spinning the cpu - provided there is a fast method
for being called on every timer tick.

Another scheme I have used is to write each value
to the bit-bang register multiple times - using
the maximum speed of the IO writes to generate the
required delays.

	David

Re: bonding with 3c59x driver

[Steffen Klassert]

On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:
> 
> Well I am still curious why the 3c59x driver has such slow polling when 
> other drivers I have been testing are able to report almost instantly 
> when I remove a network cable. Could it be that other network chips 
> generate an interrupt on cable removal and the 3com chips do not?
> 

Yes, at least some of the supported chips do not generate an interrupt
on cable removal, so we have to check for this with a timer.

Re: bonding with 3c59x driver

[Eric Dumazet]

Le mardi 14 février 2012 à 12:06 +0100, Steffen Klassert a écrit :
> On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:
> > 
> > Well I am still curious why the 3c59x driver has such slow polling when 
> > other drivers I have been testing are able to report almost instantly 
> > when I remove a network cable. Could it be that other network chips 
> > generate an interrupt on cable removal and the 3com chips do not?
> > 
> 
> Yes, at least some of the supported chips do not generate an interrupt
> on cable removal, so we have to check for this with a timer.
> --

We could have a 5 sec timer in case device is a slave.

diff --git a/drivers/net/ethernet/3com/3c59x.c b/drivers/net/ethernet/3com/3c59x.c
index 1282f04..e463d10 100644
--- a/drivers/net/ethernet/3com/3c59x.c
+++ b/drivers/net/ethernet/3com/3c59x.c
@@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
 		ok = 1;
 	}
 
-	if (!netif_carrier_ok(dev))
+	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
 		next_tick = 5*HZ;
 
 	if (vp->medialock)

Re: bonding with 3c59x driver

[Jean Delvare]

Hi Eric, Steffen,

On Tuesday 14 February 2012 12:13:37 pm Eric Dumazet wrote:
> Le mardi 14 février 2012 à 12:06 +0100, Steffen Klassert a écrit :
> > On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:
> > > Well I am still curious why the 3c59x driver has such slow
> > > polling when other drivers I have been testing are able to report
> > > almost instantly when I remove a network cable. Could it be that
> > > other network chips generate an interrupt on cable removal and
> > > the 3com chips do not?
> >
> > Yes, at least some of the supported chips do not generate an
> > interrupt on cable removal, so we have to check for this with a
> > timer.

Too bad. I confirm I can't see the interrupt count increase when I 
remove the network cable from my 3C905C (nor when I plug it back in), 
but I was hopping this was a matter of enabling this event as an 
interrupt source in the driver. If the hardware just can't do it then 
this is hopeless. Back in the days I used to consider the 3C905 as the 
best fast Ethernet cards around, apparently I was horribly wrong.

Thanks for explaining anyway.

> We could have a 5 sec timer in case device is a slave.
> 
> diff --git a/drivers/net/ethernet/3com/3c59x.c
>  b/drivers/net/ethernet/3com/3c59x.c index 1282f04..e463d10 100644
> --- a/drivers/net/ethernet/3com/3c59x.c
> +++ b/drivers/net/ethernet/3com/3c59x.c
> @@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
>  		ok = 1;
>  	}
> 
> -	if (!netif_carrier_ok(dev))
> +	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
>  		next_tick = 5*HZ;
> 
>  	if (vp->medialock)

Yes I think this makes a lot of sense. If you send this upstream you can 
add:

Acked-by: Jean Delvare <jdelvare@suse.de>

-- 
Jean Delvare
Suse L3

Re: bonding with 3c59x driver

[Steffen Klassert]

On Tue, Feb 14, 2012 at 01:50:57PM +0100, Jean Delvare wrote:
> 
> > We could have a 5 sec timer in case device is a slave.
> > 
> > diff --git a/drivers/net/ethernet/3com/3c59x.c
> >  b/drivers/net/ethernet/3com/3c59x.c index 1282f04..e463d10 100644
> > --- a/drivers/net/ethernet/3com/3c59x.c
> > +++ b/drivers/net/ethernet/3com/3c59x.c
> > @@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
> >  		ok = 1;
> >  	}
> > 
> > -	if (!netif_carrier_ok(dev))
> > +	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
> >  		next_tick = 5*HZ;
> > 
> >  	if (vp->medialock)
> 
> Yes I think this makes a lot of sense. If you send this upstream you can 
> add:
> 
> Acked-by: Jean Delvare <jdelvare@suse.de>
> 

I'm fine with this too.

Re: bonding with 3c59x driver

[Rick Jones]

On 02/14/2012 03:13 AM, Eric Dumazet wrote:
> Le mardi 14 février 2012 à 12:06 +0100, Steffen Klassert a écrit :
>> On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:
>>>
>>> Well I am still curious why the 3c59x driver has such slow polling when
>>> other drivers I have been testing are able to report almost instantly
>>> when I remove a network cable. Could it be that other network chips
>>> generate an interrupt on cable removal and the 3com chips do not?
>>>
>>
>> Yes, at least some of the supported chips do not generate an interrupt
>> on cable removal, so we have to check for this with a timer.
>> --
>
> We could have a 5 sec timer in case device is a slave.
>
> diff --git a/drivers/net/ethernet/3com/3c59x.c b/drivers/net/ethernet/3com/3c59x.c
> index 1282f04..e463d10 100644
> --- a/drivers/net/ethernet/3com/3c59x.c
> +++ b/drivers/net/ethernet/3com/3c59x.c
> @@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
>   		ok = 1;
>   	}
>
> -	if (!netif_carrier_ok(dev))
> +	if (dev->flags&  IFF_SLAVE || !netif_carrier_ok(dev))
>   		next_tick = 5*HZ;
>
>   	if (vp->medialock)

Does the device being a slave in a bond change the overheads of the 
check being performed?

rick jones

Re: bonding with 3c59x driver

[Jean Delvare]

On Tuesday 14 February 2012 07:27:05 pm Rick Jones wrote:
> On 02/14/2012 03:13 AM, Eric Dumazet wrote:
> > Le mardi 14 février 2012 à 12:06 +0100, Steffen Klassert a écrit :
> >> On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:
> >>> Well I am still curious why the 3c59x driver has such slow
> >>> polling when other drivers I have been testing are able to report
> >>> almost instantly when I remove a network cable. Could it be that
> >>> other network chips generate an interrupt on cable removal and
> >>> the 3com chips do not?
> >>
> >> Yes, at least some of the supported chips do not generate an
> >> interrupt on cable removal, so we have to check for this with a
> >> timer. --
> >
> > We could have a 5 sec timer in case device is a slave.
> >
> > diff --git a/drivers/net/ethernet/3com/3c59x.c
> > b/drivers/net/ethernet/3com/3c59x.c index 1282f04..e463d10 100644
> > --- a/drivers/net/ethernet/3com/3c59x.c
> > +++ b/drivers/net/ethernet/3com/3c59x.c
> > @@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
> >   		ok = 1;
> >   	}
> >
> > -	if (!netif_carrier_ok(dev))
> > +	if (dev->flags&  IFF_SLAVE || !netif_carrier_ok(dev))
> >   		next_tick = 5*HZ;
> >
> >   	if (vp->medialock)
> 
> Does the device being a slave in a bond change the overheads of the
> check being performed?

No, but it changes the interest in having faster polling. The overhead 
is the same but the interest/overhead trade-off is much higher.

Without the patch above, doing bonding on top of 3c59x is simply not 
possible, at least not in active-backup mode. For other modes it depends 
if you are only interested in load balancing or also on redundancy.

-- 
Jean Delvare
Suse L3

Re: bonding with 3c59x driver

[David Miller]

From: Steffen Klassert <klassert@mathematik.tu-chemnitz.de>
Date: Tue, 14 Feb 2012 14:34:18 +0100

> On Tue, Feb 14, 2012 at 01:50:57PM +0100, Jean Delvare wrote:
>> 
>> > We could have a 5 sec timer in case device is a slave.
>> > 
>> > diff --git a/drivers/net/ethernet/3com/3c59x.c
>> >  b/drivers/net/ethernet/3com/3c59x.c index 1282f04..e463d10 100644
>> > --- a/drivers/net/ethernet/3com/3c59x.c
>> > +++ b/drivers/net/ethernet/3com/3c59x.c
>> > @@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
>> >  		ok = 1;
>> >  	}
>> > 
>> > -	if (!netif_carrier_ok(dev))
>> > +	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
>> >  		next_tick = 5*HZ;
>> > 
>> >  	if (vp->medialock)
>> 
>> Yes I think this makes a lot of sense. If you send this upstream you can 
>> add:
>> 
>> Acked-by: Jean Delvare <jdelvare@suse.de>
>> 
> 
> I'm fine with this too.

Eric, please submit this formally.

[PATCH] 3c59x: shorten timer period for slave devices

[Eric Dumazet]

Jean Delvare reported bonding on top of 3c59x adapters was not detecting
network cable removal fast enough.

3c59x indeed uses a 60 seconds timer to check link status if carrier is
on, and 5 seconds if carrier is off.

This patch reduces timer period to 5 seconds if device is a bonding
slave.

Reported-by: Jean Delvare <jdelvare@suse.de>
Acked-by: Jean Delvare <jdelvare@suse.de>
Acked-by: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
 drivers/net/ethernet/3com/3c59x.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/3com/3c59x.c b/drivers/net/ethernet/3com/3c59x.c
index 1282f04..e463d10 100644
--- a/drivers/net/ethernet/3com/3c59x.c
+++ b/drivers/net/ethernet/3com/3c59x.c
@@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
 		ok = 1;
 	}
 
-	if (!netif_carrier_ok(dev))
+	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
 		next_tick = 5*HZ;
 
 	if (vp->medialock)

Re: [PATCH] 3c59x: shorten timer period for slave devices

[Dan Williams]

On Tue, 2012-02-14 at 21:27 +0100, Eric Dumazet wrote:
> Jean Delvare reported bonding on top of 3c59x adapters was not detecting
> network cable removal fast enough.
> 
> 3c59x indeed uses a 60 seconds timer to check link status if carrier is
> on, and 5 seconds if carrier is off.
> 
> This patch reduces timer period to 5 seconds if device is a bonding
> slave.

Maybe for posterity give some rationale as to why we feel we can reduce
it to 5 seconds for slaves instead of reducing the timer period in
general?  If you weren't party to this discussion that won't be apparent
from the commit log.

Dan

> Reported-by: Jean Delvare <jdelvare@suse.de>
> Acked-by: Jean Delvare <jdelvare@suse.de>
> Acked-by: Steffen Klassert <steffen.klassert@secunet.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---
>  drivers/net/ethernet/3com/3c59x.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/3com/3c59x.c b/drivers/net/ethernet/3com/3c59x.c
> index 1282f04..e463d10 100644
> --- a/drivers/net/ethernet/3com/3c59x.c
> +++ b/drivers/net/ethernet/3com/3c59x.c
> @@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
>  		ok = 1;
>  	}
>  
> -	if (!netif_carrier_ok(dev))
> +	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
>  		next_tick = 5*HZ;
>  
>  	if (vp->medialock)
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] 3c59x: shorten timer period for slave devices

[Eric Dumazet]

Le mardi 14 février 2012 à 14:51 -0600, Dan Williams a écrit :
> On Tue, 2012-02-14 at 21:27 +0100, Eric Dumazet wrote:
> > Jean Delvare reported bonding on top of 3c59x adapters was not detecting
> > network cable removal fast enough.
> > 
> > 3c59x indeed uses a 60 seconds timer to check link status if carrier is
> > on, and 5 seconds if carrier is off.
> > 
> > This patch reduces timer period to 5 seconds if device is a bonding
> > slave.
> 
> Maybe for posterity give some rationale as to why we feel we can reduce
> it to 5 seconds for slaves instead of reducing the timer period in
> general?  If you weren't party to this discussion that won't be apparent
> from the commit log.

Apparently, Andrew Morton considered firing a timer every 5 seconds was
too expensive. I am not sure we want to state this (probably old fact)
in the changelog. Anyway are these slow NICS still used in 2011 ? ;)

Re: bonding with 3c59x driver

[Chris Friesen]

On 02/14/2012 01:27 PM, Jean Delvare wrote:

> Without the patch above, doing bonding on top of 3c59x is simply not
> possible, at least not in active-backup mode. For other modes it depends
> if you are only interested in load balancing or also on redundancy.

Could you just use arp monitoring instead of mii?

Chris

-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

Re: [PATCH] 3c59x: shorten timer period for slave devices

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 14 Feb 2012 22:02:56 +0100

> Le mardi 14 février 2012 à 14:51 -0600, Dan Williams a écrit :
>> On Tue, 2012-02-14 at 21:27 +0100, Eric Dumazet wrote:
>> > Jean Delvare reported bonding on top of 3c59x adapters was not detecting
>> > network cable removal fast enough.
>> > 
>> > 3c59x indeed uses a 60 seconds timer to check link status if carrier is
>> > on, and 5 seconds if carrier is off.
>> > 
>> > This patch reduces timer period to 5 seconds if device is a bonding
>> > slave.
>> 
>> Maybe for posterity give some rationale as to why we feel we can reduce
>> it to 5 seconds for slaves instead of reducing the timer period in
>> general?  If you weren't party to this discussion that won't be apparent
>> from the commit log.
> 
> Apparently, Andrew Morton considered firing a timer every 5 seconds was
> too expensive. I am not sure we want to state this (probably old fact)
> in the changelog. Anyway are these slow NICS still used in 2011 ? ;)

At least Jean Delvare is :-)

I'm going to apply this as-is, if you configure bonding on one of these
chips you expect it to work.  And working is more important than optimal.

Re: bonding with 3c59x driver

[Jean Delvare]

On Tuesday 14 February 2012 10:06:54 pm Chris Friesen wrote:
> On 02/14/2012 01:27 PM, Jean Delvare wrote:
> > Without the patch above, doing bonding on top of 3c59x is simply
> > not possible, at least not in active-backup mode. For other modes
> > it depends if you are only interested in load balancing or also on
> > redundancy.
> 
> Could you just use arp monitoring instead of mii?

If it was just for me, I could, and probably would. But when I am trying 
to reproduce a customer bug, I have to use exactly the same options the 
customer is using.

-- 
Jean Delvare
Suse L3