Re: memory problem with fec on 8250

Re: memory problem with fec on 8250

[Samo Pogacnik]

Hello!

Is it possible that there is something wrong in the way how the:
"netif_wake_queue(dev)" gets called from the interrupt handler in the
"fcc_enet.c"?
I can confirm this problem for 2.4.18 as well as 2.6.0 kernels on mpc82xx.

bye, Samo

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Wojciech Kromer]

Użytkownik Samo Pogacnik napisał:

>Hello!
>
>Is it possible that there is something wrong in the way how the:
>"netif_wake_queue(dev)" gets called from the interrupt handler in the
>"fcc_enet.c"?
>I can confirm this problem for 2.4.18 as well as 2.6.0 kernels on mpc82xx.
>
>bye, Samo
>
>
>
>
i don't know exactly what's wrong with 82xx code,
but i wrote a small patch that frees unfreed skbuf before allocating new
one,
and now it works fine

additionaly, i had no problem with (almost) the same code on 8xx

--
* * * * * * * * * * * *
* per pedes ad astra! *
* * * * * * * * * * * *    mailto:krom@dgt-lab.com.pl


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Samo Pogacnik]

On Monday 05 January 2004 09:15, Wojciech Kromer wrote:
> Użytkownik Samo Pogacnik napisał:
> >
> >Is it possible that there is something wrong in the way how the:
> >"netif_wake_queue(dev)" gets called from the interrupt handler in the
> >"fcc_enet.c"? I can confirm this problem for 2.4.18 as well as 2.6.0
> >kernels on mpc82xx.
>
> i don't know exactly what's wrong with 82xx code, but i wrote a small
> patch that frees unfreed skbuf before allocating new one, and now it
> works fine
>
> additionaly, i had no problem with (almost) the same code on 8xx

hi, Wojciech
can you send the patch?
did you free skbuffs from the irq handler or somewhere else?

when i run netperf (snapshot) test using the 100Mbps connection, this
test eats all the available RAM. the /proc/meminfo shows that most of the used
memory at that point is been held by internal kernel structures (Slab).

by, Samo

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

cd

[Wojciech Kromer]

Użytkownik Samo Pogacnik napisał:

>On Monday 05 January 2004 09:15, Wojciech Kromer wrote:
>
>
>>Użytkownik Samo Pogacnik napisał:
>>
>>
>>>Is it possible that there is something wrong in the way how the:
>>>"netif_wake_queue(dev)" gets called from the interrupt handler in the
>>>"fcc_enet.c"? I can confirm this problem for 2.4.18 as well as 2.6.0
>>>kernels on mpc82xx.
>>>
>>>
>>i don't know exactly what's wrong with 82xx code, but i wrote a small
>>patch that frees unfreed skbuf before allocating new one, and now it
>>works fine
>>
>>additionaly, i had no problem with (almost) the same code on 8xx
>>
>>
>
>hi, Wojciech
>can you send the patch?
>
sure, just 4 lines addes

>did you free skbuffs from the irq handler or somewhere else?
>
>
just before allocation

>when i run netperf (snapshot) test using the 100Mbps connection, this
>test eats all the available RAM. the /proc/meminfo shows that most of the used
>memory at that point is been held by internal kernel structures (Slab).
>
>
better reults are shwn with
grep skb /proc/slabinfo

>by, Samo
>
>
here it is:

--------------------------------------------------------------------------------------------------------
***************
*** 413,418 ****
--- 412,420 ----

      /* Save skb pointer.
      */
+     //by k@m 2003-12-05
+     if(cep->tx_skbuff[cep->skb_cur])
+      dev_kfree_skb_irq (cep->tx_skbuff[cep->skb_cur]);
      cep->tx_skbuff[cep->skb_cur] = skb;

      cep->stats.tx_bytes += skb->len;
***************
*** 560,565 ****
--- 562,568 ----
          /* Free the sk buffer associated with this last transmit.
          */
          dev_kfree_skb_irq(cep->tx_skbuff[cep->skb_dirty]);
+         cep->tx_skbuff[cep->skb_dirty]=0;//by k@m 2003-12-05
          cep->skb_dirty = (cep->skb_dirty + 1) & TX_RING_MOD_MASK;

          /* Update pointer to next buffer descriptor to be transmitted.
***************
*** 1348,1353 ****
--- 1351,1357 ----
       * It works when I do this.
       */
      memset((char *)ep, 0, sizeof(fcc_enet_t));
+     memset(cep->tx_skbuff,0,sizeof(cep->tx_skbuff));

      /* Allocate space for the buffer descriptors in the DP ram.
       * These are relative offsets in the DP ram address space.
-------------------------------------------------------------------------------------------

ps: i send you my full version of fcc_enet.c

--
* * * * * * * * * * * *
* per pedes ad astra! *
* * * * * * * * * * * *    mailto:krom@dgt-lab.com.pl


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Samo Pogacnik]

On Tuesday 06 January 2004 08:47, Wojciech Kromer wrote:
> sure, just 4 lines addes
>
> >did you free skbuffs from the irq handler or somewhere else?
>
> just before allocation
>
> >when i run netperf (snapshot) test using the 100Mbps connection, this
> >test eats all the available RAM. the /proc/meminfo shows that most
> >of the used memory at that point is been held by internal kernel
> >structures (Slab).
>
> better reults are shwn with grep skb /proc/slabinfo
>
> here it is:
	.
	.
	.
Thank you for the tips and the patch.
Eth seems to be working now without eating the RAM.

By the way that you have solved the problem, I suspect that BDs are written
over the dirty_tx limit, before their associated skbs get freed via the irq
handler.

bye, Samo

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Dan Malek]

Wojciech Kromer wrote:

> _anyone_ knows why there is such difference between allocs and frees in
> fcc_enet.c

What kernel and driver is this?

The latest one I checked into linuxppc_2_4_devel a long time ago
had the benefit of being tested at the UNH Interoperability Laboratory.
This included performance testing and error recovery.  For this driver
to be failing, it must have been modified from that version and we
need to determine what was done to get it corrected.

Thanks.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Wojciech Kromer]

Użytkownik Eugene Surovegin napisał:

>On Thu, Dec 04, 2003 at 11:37:28AM +0100, Wojciech Kromer wrote:
>
>
>>i have to check end-of-trasmit interrupt code, but it seems to be good,
>>and it's common used driver (i hope so)
>>
>>
>
>Yep, quick look didn't reveal anything suspicious.
>
>To quickly check TX skb leak theory I'd add one counter and
>  1) increment it in fcc_start_xmit, and increment it for each
>     transmitted packet (e.g. at line 450)
>
>  2) decrement it in fcc_enet_interrupt right after dev_kfree_skb_irq
>     (at line 593).
>
>Make this counter accessible from user mode, e.g. through /proc and
>run your test :)
>
>
>
so i made such test, theese are results:

bash-2.05# cat /proc/interrupts
           CPU0
 19:        157   8260 SIU   Edge      dsp_msg
 33:      81139   8260 SIU   Edge      fenet
 34:          0   8260 SIU   Edge      fenet
 40:       3909   8260 SIU   Edge      uart
 42:        134   8260 SIU   Edge      uart
 43:          0   8260 SIU   Edge      uart
BAD:      14481

bash-2.05# cat /proc/kdebug
tx_skb_alloc=53384 tx_skb_free=29058

_anyone_ knows why there is such difference between allocs and frees in
fcc_enet.c



--
* * * * * * * * * * * *
* per pedes ad astra! *
* * * * * * * * * * * *    mailto:krom@dgt-lab.com.pl


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Eugene Surovegin]

On Thu, Dec 04, 2003 at 10:45:54AM +0100, Wojciech Kromer wrote:
> >
> >>>i have some problems with using fec device on my custom 8250 board
> >>>while sending a lot of data (at least 300MB) through this device causes
> >>>lack of memory
> >>>sending same amount of data through loopback works fine
> >>>

[snip]

> ==== at begining =========================================
> skbuff_head_cache    131    192    160    8    8    1
> size-2048              5    178   2048    3   89    1
>
> ===after some time (ftp put ~75MB) ================================
>
> skbuff_head_cache  16857  16872    160  703  703    1
> size-2048          16749  16798   2048 8382 8399    1

It looks like skb leaks in the driver. Probably in TX path.

Could you send me by private e-mail source code for ethernet driver in
question or provide a URL.

I'm not a 8250 guy, but I'll try to look at it :)

Eugene.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Wojciech Kromer]

Użytkownik Eugene Surovegin napisał:

>On Thu, Dec 04, 2003 at 09:45:53AM +0100, Wojciech Kromer wrote:
>
>
>>>i have some problems with using fec device on my custom 8250 board
>>>while sending a lot of data (at least 300MB) through this device causes
>>>lack of memory
>>>sending same amount of data through loopback works fine
>>>
>>>i'm using 2.4.19 kernel
>>>any idea?
>>>
>>>
>>cached memory keeps growing slowly, and free is going down same way,
>>after putting ~300MB ftp data (from 8250 board to another host), VM
>>starts killing some applications
>>
>>same situation is with latest linuxppc_2_4_devel kernel from denx.de,
>>but there is additional information from kernel
>>
>><5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
>><5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
>><5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
>><5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
>><5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
>><4>VM: killing process sshd
>>
>>what's wrong ???
>>
>>
>
>It means that this box is really out of memory.
>
>
sure, but _why_? thre are no memory consuption processes,
i've just run ftp-put from this box to another machine

>Could you provide output from utility called 'free' and from file
>/proc/slabinfo at two points of time: 1) when you just booted your
>boax and 2) just before it starts failing.
>
>
>
thx for quick answer

huh, a lot of info:

==== at begining =========================================
bash-2.05# free
             total       used       free     shared    buffers     cached
Mem:        127888      22488     105400          0          0      15272
-/+ buffers/cache:       7216     120672
Swap:            0          0          0
bash-2.05# cat /proc/slabinfo
slabinfo - version: 1.1
kmem_cache            62     68    112    2    2    1
tcp_tw_bucket          0      0     96    0    0    1
tcp_bind_bucket        3    113     32    1    1    1
tcp_open_request       0     59     64    0    1    1
inet_peer_cache        2     59     64    1    1    1
ip_fib_hash           10    113     32    1    1    1
ip_dst_cache          27     48    160    2    2    1
arp_cache              4     30    128    1    1    1
blkdev_requests        0      0     96    0    0    1
nfs_write_data         0     10    384    0    1    1
nfs_read_data          0     40    384    0    4    1
nfs_page               0     40     96    0    1    1
devfsd_event           0      0     20    0    0    1
journal_head           0      0     48    0    0    1
revoke_table           0      0     12    0    0    1
revoke_record          0      0     32    0    0    1
dnotify_cache          0      0     20    0    0    1
file_lock_cache        2     40     96    1    1    1
fasync_cache           0      0     16    0    0    1
uid_cache              0      0     32    0    0    1
skbuff_head_cache    131    192    160    8    8    1
sock                  21     25    800    5    5    1
sigqueue               0     29    132    0    1    1
kiobuf                 0      0     64    0    0    1
cdev_cache            10     59     64    1    1    1
bdev_cache             0      0     64    0    0    1
mnt_cache             11     59     64    1    1    1
inode_cache          419    424    480   53   53    1
dentry_cache         601    630    128   21   21    1
filp                 175    180    128    6    6    1
names_cache            0      3   4096    0    3    1
buffer_head            0      0     96    0    0    1
mm_struct             14     24    160    1    1    1
vm_area_struct       462    560     96   12   14    1
fs_cache              13    113     32    1    1    1
files_cache           14     18    416    2    2    1
signal_act            16     21   1312    6    7    1
size-131072(DMA)       0      0 131072    0    0   32
size-131072            0      0 131072    0    0   32
size-65536(DMA)        0      0  65536    0    0   16
size-65536             0      0  65536    0    0   16
size-32768(DMA)        0      0  32768    0    0    8
size-32768             0      0  32768    0    0    8
size-16384(DMA)        0      0  16384    0    0    4
size-16384             0      0  16384    0    0    4
size-8192(DMA)         0      0   8192    0    0    2
size-8192              0      0   8192    0    0    2
size-4096(DMA)         0      0   4096    0    0    1
size-4096             10     10   4096   10   10    1
size-2048(DMA)         0      0   2048    0    0    1
size-2048              5    178   2048    3   89    1
size-1024(DMA)         0      0   1024    0    0    1
size-1024             17     24   1024    5    6    1
size-512(DMA)          0      0    512    0    0    1
size-512               6      8    512    1    1    1
size-256(DMA)          0      0    256    0    0    1
size-256              14     30    256    1    2    1
size-128(DMA)          0      0    128    0    0    1
size-128             829    930    128   31   31    1
size-64(DMA)           0      0     64    0    0    1
size-64               41     59     64    1    1    1
size-32(DMA)           0      0     32    0    0    1
size-32              201    226     32    2    2    1

===after some time (ftp put ~75MB) ================================

bash-2.05# free;cat /proc/slabinfo
             total       used       free     shared    buffers     cached
Mem:        127888     125688       2200          0          0      81772
-/+ buffers/cache:      43916      83972
Swap:            0          0          0
slabinfo - version: 1.1
kmem_cache            62     68    112    2    2    1
tcp_tw_bucket          0      0     96    0    0    1
tcp_bind_bucket        5    113     32    1    1    1
tcp_open_request       0      0     64    0    0    1
inet_peer_cache        1     59     64    1    1    1
ip_fib_hash           10    113     32    1    1    1
ip_dst_cache          51     72    160    3    3    1
arp_cache              3     30    128    1    1    1
blkdev_requests        0      0     96    0    0    1
nfs_write_data         0      0    384    0    0    1
nfs_read_data          7     70    384    4    7    1
nfs_page               7     80     96    2    2    1
devfsd_event           0      0     20    0    0    1
journal_head           0      0     48    0    0    1
revoke_table           0      0     12    0    0    1
revoke_record          0      0     32    0    0    1
dnotify_cache          0      0     20    0    0    1
file_lock_cache        2     40     96    1    1    1
fasync_cache           0      0     16    0    0    1
uid_cache              0      0     32    0    0    1
skbuff_head_cache  16857  16872    160  703  703    1
sock                  23     25    800    5    5    1
sigqueue               0     29    132    0    1    1
kiobuf                 0      0     64    0    0    1
cdev_cache            10     59     64    1    1    1
bdev_cache             0      0     64    0    0    1
mnt_cache             11     59     64    1    1    1
inode_cache          435    440    480   55   55    1
dentry_cache         625    630    128   21   21    1
filp                 179    180    128    6    6    1
names_cache            0      2   4096    0    2    1
buffer_head            0      0     96    0    0    1
mm_struct             15     24    160    1    1    1
vm_area_struct       485    520     96   13   13    1
fs_cache              14    113     32    1    1    1
files_cache           15     18    416    2    2    1
signal_act            17     18   1312    6    6    1
size-131072(DMA)       0      0 131072    0    0   32
size-131072            0      0 131072    0    0   32
size-65536(DMA)        0      0  65536    0    0   16
size-65536             0      0  65536    0    0   16
size-32768(DMA)        0      0  32768    0    0    8
size-32768             0      0  32768    0    0    8
size-16384(DMA)        0      0  16384    0    0    4
size-16384             0      0  16384    0    0    4
size-8192(DMA)         0      0   8192    0    0    2
size-8192              0      0   8192    0    0    2
size-4096(DMA)         0      0   4096    0    0    1
size-4096             10     10   4096   10   10    1
size-2048(DMA)         0      0   2048    0    0    1
size-2048          16749  16798   2048 8382 8399    1
size-1024(DMA)         0      0   1024    0    0    1
size-1024             18     20   1024    5    5    1
size-512(DMA)          0      0    512    0    0    1
size-512              24     32    512    4    4    1
size-256(DMA)          0      0    256    0    0    1
size-256              51     75    256    4    5    1
size-128(DMA)          0      0    128    0    0    1
size-128             910    930    128   31   31    1
size-64(DMA)           0      0     64    0    0    1
size-64               42     59     64    1    1    1
size-32(DMA)           0      0     32    0    0    1
size-32             8513   8588     32   76   76    1

--
* * * * * * * * * * * *
* per pedes ad astra! *
* * * * * * * * * * * *    mailto:krom@dgt-lab.com.pl


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Eugene Surovegin]

On Thu, Dec 04, 2003 at 09:45:53AM +0100, Wojciech Kromer wrote:
> >i have some problems with using fec device on my custom 8250 board
> >while sending a lot of data (at least 300MB) through this device causes
> >lack of memory
> >sending same amount of data through loopback works fine
> >
> >i'm using 2.4.19 kernel
> >any idea?
>
> cached memory keeps growing slowly, and free is going down same way,
> after putting ~300MB ftp data (from 8250 board to another host), VM
> starts killing some applications
>
> same situation is with latest linuxppc_2_4_devel kernel from denx.de,
> but there is additional information from kernel
>
> <5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> <5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> <5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> <5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> <5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
> <4>VM: killing process sshd
>
> what's wrong ???

It means that this box is really out of memory.

Could you provide output from utility called 'free' and from file
/proc/slabinfo at two points of time: 1) when you just booted your
boax and 2) just before it starts failing.

Thanks,

Eugene.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: memory problem with fec on 8250

[Wojciech Kromer]

Użytkownik Kromer napisał:

>
> hi there
>
> i have some problems with using fec device on my custom 8250 board
> while sending a lot of data (at least 300MB) through this device causes
> lack of memory
> sending same amount of data through loopback works fine
>
> i'm using 2.4.19 kernel
> any idea?

cached memory keeps growing slowly, and free is going down same way,
after putting ~300MB ftp data (from 8250 board to another host), VM
starts killing some applications

same situation is with latest linuxppc_2_4_devel kernel from denx.de,
but there is additional information from kernel

<5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
<5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
<5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
<5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
<5>__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
<4>VM: killing process sshd

what's wrong ???


--
* * * * * * * * * * * *
* per pedes ad astra! *
* * * * * * * * * * * *    mailto:krom@dgt-lab.com.pl


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

memory problem with fec on 8250

[Kromer]

hi there

i have some problems with using fec device on my custom 8250 board
while sending a lot of data (at least 300MB) through this device causes
lack of memory
sending same amount of data through loopback works fine

i'm using 2.4.19 kernel
any idea?


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/