* Re: fec driver question
2009-04-29 10:01 ` fec driver question Philippe De Muyter
@ 2009-04-29 11:11 ` Matthew Lear
2009-04-30 0:43 ` [uClinux-dev] " Greg Ungerer
1 sibling, 0 replies; 3+ messages in thread
From: Matthew Lear @ 2009-04-29 11:11 UTC (permalink / raw)
To: Philippe De Muyter; +Cc: netdev, uclinux-dev
> Hi Matthew,
>
> [CCing uclinux-dev@uclinux.org and netdev@vger.kernel.org]
>
> On Wed, Apr 29, 2009 at 09:48:37AM +0100, Matthew Lear wrote:
>> Hi Phillippe - Thanks very much for your reply. some comments below:
>>
>> > Hi Matthew,
>> > On Wed, Apr 29, 2009 at 08:15:43AM +0100, Matthew Lear wrote:
>> >> Hello Philippe,
>> >>
>> >> I hope you don't mind me emailing you. Basically I have a dev board
>> from
>> >> freescale for doing some coldfire development on the mfc54455 device.
>> >> I'm
>> >> using the fec driver in the kernel. Kernel version is 2.6.23. I'm
>> havng
>> >> some problems and I was hoping you might be able to help me.
>> >>
>> >> It seems that running some quite heavy network throughput tests on
>> the
>> >> platform result in the driver dropping packets and the userspace app
>> >> running on the dev board to run consume ~ 85% cpu. I'm using netcat
>> as
>> >> the
>> >> app running on host and the target to do the tests.
>> >>
>> >> I can appreciate that this question is somewhat 'open' in that there
>> >> could be several causes but I'm fairly certain that a) it's not
>> ksoftirq
>> >> related and b) that it's not driver related (because it's mature and
>> has
>> >> been used in all sorts of different application/platforms).
>> >>
>> >> Can you think of any possible causes for this? The fact that the
>> driver
>> >> is
>> >> dropping packets is surely indicative of there not being enough
>> buffers
>> >> to
>> >> place the incoming data and/or there are issues with the consumption
>> >> (and
>> >> subsequent freeing of these buffers) by something else.
>> >
>> > 1. You could make the same test after increasing the number of receive
>> > buffers
>> > in the driver.
>> >
>> > 2. Actually, each incoming packet generates one interrupt so it needs
>> some
>> > processing time in the interrupt service routine. Hence if your
>> receive
>> > app itself consumes 85% CPU that's probably normal that at times all
>> > buffers
>> > are used and that the chip has to drop frames. Check if you have idle
>> > time
>> > remaining.
>> >
>> > 3. It can also be a hardware bug/limitation in the chip itself. I
>> used
>> > mainly the FEC driver with mcf5272 chips at 10 Mbps, because 100 Mbps
>> > was not really supported in hardware, although it was possible to ask
>> for
>> > it.
>> > There is an offical errata for that :)
>>
>> I did try to increase the number of buffers and I was surprised at the
>> result because it seemed that the cpu utilisation of the user space app
>> increases. There are some comments at the top of fec.c regarding keeping
>> numbers associated to the buffers as powers of 2. I increased the number
>> of buffers to 32 but bizarrely it seemed to make things worse (netcat
>> consumed ~ 95% cpu). Not sure what's going on there!
>
> For me, it means that you loose/drop less packets. I surmise that your
> CPU is mmu-less, so packets must be copied from kernel to userspace for
> each received packet. That time passed by the kernel in copying the
> packet for the app is counted as app time, I presume.
> You could measure memcpy's speed and compute how much time is needed
> with your expected throughput.
Apologies Philippe. I should have been more clear. The cpu has an mmu. I'm
running Freescale not uClinux.
>>
>> When you say "check idle time remaining", do you mean in the driver
>> itself
>> or use a profiling tool?
>
> I only meant looking for %id in 'top' header.
>
>>
>> I have seen the scenario of the cpu at ~85% and no packets dropped but
>> typically there are overruns and in this case /proc/net/dev indicates
>> that
>> there are fifo issues within the driver somehow.
>>
>> Yes. One interrupt per packet is what I expected but I also have an SH4
>> dev board (though it uses a different ethernet driver). Running the same
>> kernel version and exactly the same test with netcat on that platform
>> shows seriously contrasting results in that cpu utilisation of netcat on
>> the sh4 target is minimal (as it should be).
>
> Could that be that sh4 has a mmu, and that its ethernet driver implement
> zero-copy mode ? I'm not an expert in that area though.
>
>>
>> I'm suspecting that it may be a mm or dma issue with how the buffers are
>> relayed to the upper layers. The driver is mature isn't it so I would
>> have
>
> I'm not sure at all that dma is used here, but I could be wrong.
>
>> expected that any problem such as this would have been spotted long
>> before
>> now? In this regard, I am of the opinion that it could possible be an
>> issue with the device as you say.
>
> It depends on what other people do with the ethernet device on their
> board. Here it is only used for some lightweight communication.
> And, when I used it, the driver was already mature, but I still discovered
> real bugs in initialisation sequences and error recovery, e.g. when
> testing
> link connection/disconnection.
It would be interesting to hear from anybody specifically using the fec
driver on a coldfire based platform in relation to performance under heavy
network traffic conditions.
>>
>> The coldfire part I have is specified as supporting 10 and 100 Mbps so I
>> assume that there are no issues with it. Interesting though that you
>> mention the errata...
>>
>> I think it's just a case of trying to find where the cpu is spending its
>> time. It is quite frustrating though... :-(
>
> Yes, that's part of our job :)
Indeed :)
> Best regards
>
> Philippe
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [uClinux-dev] Re: fec driver question
2009-04-29 10:01 ` fec driver question Philippe De Muyter
2009-04-29 11:11 ` Matthew Lear
@ 2009-04-30 0:43 ` Greg Ungerer
1 sibling, 0 replies; 3+ messages in thread
From: Greg Ungerer @ 2009-04-30 0:43 UTC (permalink / raw)
To: uClinux development list; +Cc: Matthew Lear, netdev
Philippe De Muyter wrote:
> Hi Matthew,
>
> [CCing uclinux-dev@uclinux.org and netdev@vger.kernel.org]
>
> On Wed, Apr 29, 2009 at 09:48:37AM +0100, Matthew Lear wrote:
>> Hi Phillippe - Thanks very much for your reply. some comments below:
>>
>>> Hi Matthew,
>>> On Wed, Apr 29, 2009 at 08:15:43AM +0100, Matthew Lear wrote:
>>>> Hello Philippe,
>>>>
>>>> I hope you don't mind me emailing you. Basically I have a dev board from
>>>> freescale for doing some coldfire development on the mfc54455 device.
>>>> I'm
>>>> using the fec driver in the kernel. Kernel version is 2.6.23. I'm havng
>>>> some problems and I was hoping you might be able to help me.
>>>>
>>>> It seems that running some quite heavy network throughput tests on the
>>>> platform result in the driver dropping packets and the userspace app
>>>> running on the dev board to run consume ~ 85% cpu. I'm using netcat as
>>>> the
>>>> app running on host and the target to do the tests.
>>>>
>>>> I can appreciate that this question is somewhat 'open' in that there
>>>> could be several causes but I'm fairly certain that a) it's not ksoftirq
>>>> related and b) that it's not driver related (because it's mature and has
>>>> been used in all sorts of different application/platforms).
>>>>
>>>> Can you think of any possible causes for this? The fact that the driver
>>>> is
>>>> dropping packets is surely indicative of there not being enough buffers
>>>> to
>>>> place the incoming data and/or there are issues with the consumption
>>>> (and
>>>> subsequent freeing of these buffers) by something else.
>>> 1. You could make the same test after increasing the number of receive
>>> buffers
>>> in the driver.
>>>
>>> 2. Actually, each incoming packet generates one interrupt so it needs some
>>> processing time in the interrupt service routine. Hence if your receive
>>> app itself consumes 85% CPU that's probably normal that at times all
>>> buffers
>>> are used and that the chip has to drop frames. Check if you have idle
>>> time
>>> remaining.
>>>
>>> 3. It can also be a hardware bug/limitation in the chip itself. I used
>>> mainly the FEC driver with mcf5272 chips at 10 Mbps, because 100 Mbps
>>> was not really supported in hardware, although it was possible to ask for
>>> it.
>>> There is an offical errata for that :)
>> I did try to increase the number of buffers and I was surprised at the
>> result because it seemed that the cpu utilisation of the user space app
>> increases. There are some comments at the top of fec.c regarding keeping
>> numbers associated to the buffers as powers of 2. I increased the number
>> of buffers to 32 but bizarrely it seemed to make things worse (netcat
>> consumed ~ 95% cpu). Not sure what's going on there!
>
> For me, it means that you loose/drop less packets. I surmise that your
> CPU is mmu-less, so packets must be copied from kernel to userspace for
> each received packet. That time passed by the kernel in copying the
> packet for the app is counted as app time, I presume.
> You could measure memcpy's speed and compute how much time is needed
> with your expected throughput.
>
>> When you say "check idle time remaining", do you mean in the driver itself
>> or use a profiling tool?
>
> I only meant looking for %id in 'top' header.
>
>> I have seen the scenario of the cpu at ~85% and no packets dropped but
>> typically there are overruns and in this case /proc/net/dev indicates that
>> there are fifo issues within the driver somehow.
>>
>> Yes. One interrupt per packet is what I expected but I also have an SH4
>> dev board (though it uses a different ethernet driver). Running the same
>> kernel version and exactly the same test with netcat on that platform
>> shows seriously contrasting results in that cpu utilisation of netcat on
>> the sh4 target is minimal (as it should be).
>
> Could that be that sh4 has a mmu, and that its ethernet driver implement
> zero-copy mode ? I'm not an expert in that area though.
>
>> I'm suspecting that it may be a mm or dma issue with how the buffers are
>> relayed to the upper layers. The driver is mature isn't it so I would have
>
> I'm not sure at all that dma is used here, but I could be wrong.
>
>> expected that any problem such as this would have been spotted long before
>> now? In this regard, I am of the opinion that it could possible be an
>> issue with the device as you say.
>
> It depends on what other people do with the ethernet device on their
> board. Here it is only used for some lightweight communication.
> And, when I used it, the driver was already mature, but I still discovered
> real bugs in initialisation sequences and error recovery, e.g. when testing
> link connection/disconnection.
>
>> The coldfire part I have is specified as supporting 10 and 100 Mbps so I
>> assume that there are no issues with it. Interesting though that you
>> mention the errata...
>>
>> I think it's just a case of trying to find where the cpu is spending its
>> time. It is quite frustrating though... :-(
>
> Yes, that's part of our job :)
Profiling the kernel is relatively easy, would be a good place to
start.
Regards
Greg
------------------------------------------------------------------------
Greg Ungerer -- Principal Engineer EMAIL: gerg@snapgear.com
SnapGear Group, McAfee PHONE: +61 7 3435 2888
825 Stanley St, FAX: +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia WEB: http://www.SnapGear.com
^ permalink raw reply [flat|nested] 3+ messages in thread