All of lore.kernel.org
 help / color / mirror / Atom feed
* Kernel Panic on OOM with 10 HTB rules
@ 2010-09-08 10:52 Anand Raj Manickam
  2010-09-08 12:42 ` Eric Dumazet
  0 siblings, 1 reply; 8+ messages in thread
From: Anand Raj Manickam @ 2010-09-08 10:52 UTC (permalink / raw)
  To: netdev, netfilter-devel, shemminger

Hi,

I m struck with
Kernel Panic - Out of Memory Issue  when  traffic  is more than 100k
pps  and have  around 10 tc rules.

There is NO kernel panic with 1 tc rule .

I using a 2.6.34.6 ( latest stable kernel) with SMP enabled

The rules are simple -
10 iptable rules :
iptables -t mangle -I FORWARD -s x.x.x.x  -j CLASSIFY --set-class 0xz:0xy

10 tc rules :
tc  class add dev imq0 parent 1:z classid 1:y htb rate 2kbit ceil 32768kbps

I m using  Intel Xeon Quad core x5570, 12GB RAM , with 8 e1000e , irq
affined ( 1 pair per core)

There is NO issue / CRASH with multiple iptable rules marking to the
same tc rule/HTB class , i can sucessfully pass more than 200k pps /
800Mbps traffic .

But when each iptable rule is marking to seprate tc rule/HTB class and
when the traffic exceeds 100K pps / 20000 connections per second , we
hit this Out of Memory - Kernel panic condition .
Any help is appreciated . Do let me know if any more further
information is need.

Thanks,
Anand

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Kernel Panic on OOM with 10 HTB rules
  2010-09-08 10:52 Kernel Panic on OOM with 10 HTB rules Anand Raj Manickam
@ 2010-09-08 12:42 ` Eric Dumazet
  2010-09-08 14:09   ` Anand Raj Manickam
  2010-09-16  9:35   ` Patrick McHardy
  0 siblings, 2 replies; 8+ messages in thread
From: Eric Dumazet @ 2010-09-08 12:42 UTC (permalink / raw)
  To: Anand Raj Manickam; +Cc: netdev, netfilter-devel, shemminger

Le mercredi 08 septembre 2010 à 16:22 +0530, Anand Raj Manickam a
écrit :
> Hi,
> 
> I m struck with
> Kernel Panic - Out of Memory Issue  when  traffic  is more than 100k
> pps  and have  around 10 tc rules.
> 
> There is NO kernel panic with 1 tc rule .
> 
> I using a 2.6.34.6 ( latest stable kernel) with SMP enabled
> 
> The rules are simple -
> 10 iptable rules :
> iptables -t mangle -I FORWARD -s x.x.x.x  -j CLASSIFY --set-class 0xz:0xy
> 
> 10 tc rules :
> tc  class add dev imq0 parent 1:z classid 1:y htb rate 2kbit ceil 32768kbps
> 
> I m using  Intel Xeon Quad core x5570, 12GB RAM , with 8 e1000e , irq
> affined ( 1 pair per core)
> 
> There is NO issue / CRASH with multiple iptable rules marking to the
> same tc rule/HTB class , i can sucessfully pass more than 200k pps /
> 800Mbps traffic .
> 
> But when each iptable rule is marking to seprate tc rule/HTB class and
> when the traffic exceeds 100K pps / 20000 connections per second , we
> hit this Out of Memory - Kernel panic condition .
> Any help is appreciated . Do let me know if any more further
> information is need.

Hi Anand

Could you give :

1) A complete stack trace / panic report
2) cat /proc/interrupts
3) ifconfig -a
4) cat /proc/sys/vm/min_free_kbytes
5) cat /proc/meminfo

for each eth*
 ethtool -g eth*



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Kernel Panic on OOM with 10 HTB rules
  2010-09-08 12:42 ` Eric Dumazet
@ 2010-09-08 14:09   ` Anand Raj Manickam
  2010-09-08 14:45     ` Eric Dumazet
  2010-09-16  9:35   ` Patrick McHardy
  1 sibling, 1 reply; 8+ messages in thread
From: Anand Raj Manickam @ 2010-09-08 14:09 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: netdev, netfilter-devel, shemminger

Please find the info below  ... its a long list though

On Wed, Sep 8, 2010 at 6:12 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le mercredi 08 septembre 2010 à 16:22 +0530, Anand Raj Manickam a
> écrit :
>> Hi,
>>
>> I m struck with
>> Kernel Panic - Out of Memory Issue  when  traffic  is more than 100k
>> pps  and have  around 10 tc rules.
>>
>> There is NO kernel panic with 1 tc rule .
>>
>> I using a 2.6.34.6 ( latest stable kernel) with SMP enabled
>>
>> The rules are simple -
>> 10 iptable rules :
>> iptables -t mangle -I FORWARD -s x.x.x.x  -j CLASSIFY --set-class 0xz:0xy
>>
>> 10 tc rules :
>> tc  class add dev imq0 parent 1:z classid 1:y htb rate 2kbit ceil 32768kbps
>>
>> I m using  Intel Xeon Quad core x5570, 12GB RAM , with 8 e1000e , irq
>> affined ( 1 pair per core)
>>
>> There is NO issue / CRASH with multiple iptable rules marking to the
>> same tc rule/HTB class , i can sucessfully pass more than 200k pps /
>> 800Mbps traffic .
>>
>> But when each iptable rule is marking to seprate tc rule/HTB class and
>> when the traffic exceeds 100K pps / 20000 connections per second , we
>> hit this Out of Memory - Kernel panic condition .
>> Any help is appreciated . Do let me know if any more further
>> information is need.
>
> Hi Anand
>
> Could you give :
>
> 1) A complete stack trace / panic report

Sorry there was no stack trace :-(


e5570#Out of memory: kill process 10240 (apache2) score 58700 or a child
Killed process 10240 (apache2) vsz:234800kB, anon-rss:1576kB, file-rss:1212kB
Out of memory: kill process 10241 (apache2) score 58700 or a child
Killed process 10241 (apache2) vsz:234800kB, anon-rss:1576kB, file-rss:1232kB
Out of memory: kill process 16120 (java) score 19383 or a child
Killed process 16120 (java) vsz:1240516kB, anon-rss:161428kB, file-rss:7516kB
Out of memory: kill process 2185 (mysqld) score 16377 or a child
Killed process 2185 (mysqld) vsz:131020kB, anon-rss:15392kB, file-rss:5144kB
Out of memory: kill process 10238 (apache2) score 3630 or a child
Killed process 21304 (tcstat_total.cg) vsz:2848kB, anon-rss:192kB,
file-rss:1016kB
Out of memory: kill process 10238 (apache2) score 3273 or a child
Killed process 10238 (apache2) vsz:13092kB, anon-rss:1408kB, file-rss:624kB
Out of memory: kill process 21315 (tcstat_total.cg) score 712 or a child
Killed process 21315 (tcstat_total.cg) vsz:2848kB, anon-rss:192kB,
file-rss:196kB
Out of memory: kill process 1983 (rsyslogd) score 449 or a child
Killed process 1983 (rsyslogd) vsz:28744kB, anon-rss:784kB, file-rss:912kB
Out of memory: kill process 10236 (apache2) score 208 or a child
Killed process 10236 (apache2) vsz:13320kB, anon-rss:1412kB, file-rss:1332kB
Out of memory: kill process 4051 (login) score 77 or a child
Killed process 4831 (bash) vsz:4556kB, anon-rss:396kB, file-rss:1336kB
Out of memory: kill process 2728 (cron) score 58 or a child
Killed process 2728 (cron) vsz:3716kB, anon-rss:156kB, file-rss:628kB
Out of memory: kill process 2068 (mysqld_safe) score 45 or a child
Killed process 2068 (mysqld_safe) vsz:2904kB, anon-rss:248kB, file-rss:1084kB
Out of memory: kill process 4051 (login) score 41 or a child
Killed process 4051 (login) vsz:2680kB, anon-rss:252kB, file-rss:948kB
Out of memory: kill process 2313 (inetd) score 31 or a child
Killed process 2313 (inetd) vsz:2004kB, anon-rss:92kB, file-rss:496kB
Out of memory: kill process 4045 (getty) score 28 or a child
Killed process 4045 (getty) vsz:1816kB, anon-rss:68kB, file-rss:400kB
Out of memory: kill process 4046 (getty) score 28 or a child
Killed process 4046 (getty) vsz:1816kB, anon-rss:72kB, file-rss:400kB
Out of memory: kill process 4047 (getty) score 28 or a child
Killed process 4047 (getty) vsz:1816kB, anon-rss:72kB, file-rss:400kB
Out of memory: kill process 4048 (getty) score 28 or a child
Killed process 4048 (getty) vsz:1816kB, anon-rss:72kB, file-rss:400kB
Out of memory: kill process 4049 (getty) score 28 or a child
Killed process 4049 (getty) vsz:1816kB, anon-rss:68kB, file-rss:400kB
Out of memory: kill process 4050 (getty) score 28 or a child
Killed process 4050 (getty) vsz:1816kB, anon-rss:68kB, file-rss:400kB
Kernel panic - not syncing: Out of memory and no killable processes...


> 2) cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3
  0:        250          0          0          0   IO-APIC-edge      timer
  1:          0          0          0          2   IO-APIC-edge      i8042
  2:          0          0          0          0    XT-PIC-XT        cascade
  4:          0          0          0        226   IO-APIC-edge      serial
  8:          0          0          0          2   IO-APIC-edge      rtc0
 12:          0          0          0          3   IO-APIC-edge      i8042
 19:          0      10673          0          0   IO-APIC-fasteoi   ata_piix
 33:          0          0          0          0   PCI-MSI-edge      pciehp
 34:          0          0          0          0   PCI-MSI-edge      pciehp
 35:          0          0          0          1   PCI-MSI-edge      pciehp
 36:          1          0          0          0   PCI-MSI-edge      pciehp
 37:          1          0          0          0   PCI-MSI-edge      pciehp
 38:          0          1          0          0   PCI-MSI-edge      pciehp
 39:         94          0          0          0   PCI-MSI-edge      eth0
 40:         88          6          0          0   PCI-MSI-edge      eth1
 41:          0         88          6          0   PCI-MSI-edge      eth2
 42:          0         88          6          0   PCI-MSI-edge      eth3
 43:          0          0         92          0   PCI-MSI-edge      eth4-rx-0
 44:          0          0          0          0   PCI-MSI-edge      eth4-tx-0
 45:          3          0          0          0   PCI-MSI-edge      eth4
 46:          0          4         88          0   PCI-MSI-edge      eth5-rx-0
 47:          0          0          0          0   PCI-MSI-edge      eth5-tx-0
 48:          0          0          3          0   PCI-MSI-edge      eth5
 49:          0          0          4         88   PCI-MSI-edge      eth6-rx-0
 50:          0          0          0          0   PCI-MSI-edge      eth6-tx-0
 51:          0          0          0          3   PCI-MSI-edge      eth6
 52:          4          0          0         88   PCI-MSI-edge      eth7-rx-0
 53:          0          0          0          0   PCI-MSI-edge      eth7-tx-0
 54:          0          3          0          0   PCI-MSI-edge      eth7
NMI:          0          0          0          0   Non-maskable interrupts
LOC:     196962     205933     209322     208530   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance
monitoring interrupts
PND:          0          0          0          0   Performance pending work
RES:       4866        761        416        467   Rescheduling interrupts
CAL:         39        102        107        109   Function call interrupts
TLB:       6019       5481       4678       6759   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          2          2          2          2   Machine check polls
ERR:          3
MIS:          0



> 3) ifconfig -a
br0       Link encap:Ethernet  HWaddr 00:90:0b:14:ac:38
          inet addr:192.168.164.222  Bcast:192.168.167.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:50 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2300 (2.2 KiB)  TX bytes:0 (0.0 B)

eth0      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:38
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:10119 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20075 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6654384 (6.3 MiB)  TX bytes:1427890 (1.3 MiB)
          Memory:fb580000-fb5a0000

eth1      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:39
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:20039 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10155 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1495356 (1.4 MiB)  TX bytes:6646438 (6.3 MiB)
          Memory:fb5e0000-fb600000

eth2      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:3a
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:19190 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9727 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1432004 (1.3 MiB)  TX bytes:6365028 (6.0 MiB)
          Memory:fb680000-fb6a0000

eth3      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:3b
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:9689 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19228 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6372426 (6.0 MiB)  TX bytes:1367600 (1.3 MiB)
          Memory:fb6e0000-fb700000
eth4      Link encap:Ethernet  HWaddr 00:90:0b:15:79:18
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:9252 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6085260 (5.8 MiB)  TX bytes:1305576 (1.2 MiB)
          Memory:fb8e0000-fb900000

eth5      Link encap:Ethernet  HWaddr 00:90:0b:15:79:19
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:18316 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9290 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1366920 (1.3 MiB)  TX bytes:6078300 (5.7 MiB)
          Memory:fb9e0000-fba00000

eth6      Link encap:Ethernet  HWaddr 00:90:0b:15:79:1a
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:18709 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9484 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1396126 (1.3 MiB)  TX bytes:6204656 (5.9 MiB)
          Memory:fbae0000-fbb00000

eth7      Link encap:Ethernet  HWaddr 00:90:0b:15:79:1b
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:9446 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18746 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6211812 (5.9 MiB)  TX bytes:1333280 (1.2 MiB)
          Memory:fbbe0000-fbc00000

imq0      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          UP RUNNING NOARP  MTU:16000  Metric:1
          RX packets:129112 errors:0 dropped:0 overruns:0 frame:0
          TX packets:129114 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
          RX bytes:31060964 (29.6 MiB)  TX bytes:31062207 (29.6 MiB)

imq1      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

imq2      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

imq3      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

imq4      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

imq5      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

imq6      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
imq7      Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16000  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:11000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:14454 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14454 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1962561 (1.8 MiB)  TX bytes:1962561 (1.8 MiB)



> 4) cat /proc/sys/vm/min_free_kbytes
3806

> 5) cat /proc/meminfo

MemTotal:       12462064 kB
MemFree:        12265628 kB
Buffers:           29672 kB
Cached:            64800 kB
SwapCached:            0 kB
Active:            87748 kB
Inactive:          75756 kB
Active(anon):      69040 kB
Inactive(anon):      288 kB
Active(file):      18708 kB
Inactive(file):    75468 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:      11660872 kB
HighFree:       11515792 kB
LowTotal:         801192 kB
LowFree:          749836 kB
SwapTotal:      27226684 kB
SwapFree:       27226684 kB
Dirty:               120 kB
Writeback:             0 kB
AnonPages:         69032 kB
Mapped:            14508 kB
Shmem:               296 kB
Slab:              15728 kB
SReclaimable:       6044 kB
SUnreclaim:         9684 kB
KernelStack:        1376 kB
PageTables:          996 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    33457716 kB
Committed_AS:     848336 kB
VmallocTotal:     122880 kB
VmallocUsed:        9468 kB
VmallocChunk:     106708 kB
DirectMap4k:        6136 kB
DirectMap2M:      907264 kB


> for each eth*
>  ethtool -g eth*

e5570#ethtool -g eth0 - eth7 ( have the same config)
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       0
TX:             256
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Kernel Panic on OOM with 10 HTB rules
  2010-09-08 14:09   ` Anand Raj Manickam
@ 2010-09-08 14:45     ` Eric Dumazet
  2010-09-08 15:02       ` Krzysztof Olędzki
  2010-09-08 17:25       ` Jarek Poplawski
  0 siblings, 2 replies; 8+ messages in thread
From: Eric Dumazet @ 2010-09-08 14:45 UTC (permalink / raw)
  To: Anand Raj Manickam; +Cc: netdev, netfilter-devel, shemminger

Le mercredi 08 septembre 2010 à 19:39 +0530, Anand Raj Manickam a
écrit :
> Please find the info below  ... its a long list though

> Sorry there was no stack trace :-(
> 
> 
> e5570#Out of memory: kill process 10240 (apache2) score 58700 or a child
> Killed process 10240 (apache2) vsz:234800kB, anon-rss:1576kB, file-rss:1212kB
> Out of memory: kill process 10241 (apache2) score 58700 or a child
> Killed process 10241 (apache2) vsz:234800kB, anon-rss:1576kB, file-rss:1232kB
> Out of memory: kill process 16120 (java) score 19383 or a child
> Killed process 16120 (java) vsz:1240516kB, anon-rss:161428kB, file-rss:7516kB
> Out of memory: kill process 2185 (mysqld) score 16377 or a child
> Killed process 2185 (mysqld) vsz:131020kB, anon-rss:15392kB, file-rss:5144kB
> Out of memory: kill process 10238 (apache2) score 3630 or a child
> Killed process 21304 (tcstat_total.cg) vsz:2848kB, anon-rss:192kB,
> file-rss:1016kB
> Out of memory: kill process 10238 (apache2) score 3273 or a child
> Killed process 10238 (apache2) vsz:13092kB, anon-rss:1408kB, file-rss:624kB
> Out of memory: kill process 21315 (tcstat_total.cg) score 712 or a child
> Killed process 21315 (tcstat_total.cg) vsz:2848kB, anon-rss:192kB,
> file-rss:196kB
> Out of memory: kill process 1983 (rsyslogd) score 449 or a child
> Killed process 1983 (rsyslogd) vsz:28744kB, anon-rss:784kB, file-rss:912kB
> Out of memory: kill process 10236 (apache2) score 208 or a child
> Killed process 10236 (apache2) vsz:13320kB, anon-rss:1412kB, file-rss:1332kB
> Out of memory: kill process 4051 (login) score 77 or a child
> Killed process 4831 (bash) vsz:4556kB, anon-rss:396kB, file-rss:1336kB
> Out of memory: kill process 2728 (cron) score 58 or a child
> Killed process 2728 (cron) vsz:3716kB, anon-rss:156kB, file-rss:628kB
> Out of memory: kill process 2068 (mysqld_safe) score 45 or a child
> Killed process 2068 (mysqld_safe) vsz:2904kB, anon-rss:248kB, file-rss:1084kB
> Out of memory: kill process 4051 (login) score 41 or a child
> Killed process 4051 (login) vsz:2680kB, anon-rss:252kB, file-rss:948kB
> Out of memory: kill process 2313 (inetd) score 31 or a child
> Killed process 2313 (inetd) vsz:2004kB, anon-rss:92kB, file-rss:496kB
> Out of memory: kill process 4045 (getty) score 28 or a child
> Killed process 4045 (getty) vsz:1816kB, anon-rss:68kB, file-rss:400kB
> Out of memory: kill process 4046 (getty) score 28 or a child
> Killed process 4046 (getty) vsz:1816kB, anon-rss:72kB, file-rss:400kB
> Out of memory: kill process 4047 (getty) score 28 or a child
> Killed process 4047 (getty) vsz:1816kB, anon-rss:72kB, file-rss:400kB
> Out of memory: kill process 4048 (getty) score 28 or a child
> Killed process 4048 (getty) vsz:1816kB, anon-rss:72kB, file-rss:400kB
> Out of memory: kill process 4049 (getty) score 28 or a child
> Killed process 4049 (getty) vsz:1816kB, anon-rss:68kB, file-rss:400kB
> Out of memory: kill process 4050 (getty) score 28 or a child
> Killed process 4050 (getty) vsz:1816kB, anon-rss:68kB, file-rss:400kB
> Kernel panic - not syncing: Out of memory and no killable processes...
> 
> 
> > 2) cat /proc/interrupts
> 
>            CPU0       CPU1       CPU2       CPU3
>   0:        250          0          0          0   IO-APIC-edge      timer
>   1:          0          0          0          2   IO-APIC-edge      i8042
>   2:          0          0          0          0    XT-PIC-XT        cascade
>   4:          0          0          0        226   IO-APIC-edge      serial
>   8:          0          0          0          2   IO-APIC-edge      rtc0
>  12:          0          0          0          3   IO-APIC-edge      i8042
>  19:          0      10673          0          0   IO-APIC-fasteoi   ata_piix
>  33:          0          0          0          0   PCI-MSI-edge      pciehp
>  34:          0          0          0          0   PCI-MSI-edge      pciehp
>  35:          0          0          0          1   PCI-MSI-edge      pciehp
>  36:          1          0          0          0   PCI-MSI-edge      pciehp
>  37:          1          0          0          0   PCI-MSI-edge      pciehp
>  38:          0          1          0          0   PCI-MSI-edge      pciehp
>  39:         94          0          0          0   PCI-MSI-edge      eth0
>  40:         88          6          0          0   PCI-MSI-edge      eth1
>  41:          0         88          6          0   PCI-MSI-edge      eth2
>  42:          0         88          6          0   PCI-MSI-edge      eth3
>  43:          0          0         92          0   PCI-MSI-edge      eth4-rx-0
>  44:          0          0          0          0   PCI-MSI-edge      eth4-tx-0
>  45:          3          0          0          0   PCI-MSI-edge      eth4
>  46:          0          4         88          0   PCI-MSI-edge      eth5-rx-0
>  47:          0          0          0          0   PCI-MSI-edge      eth5-tx-0
>  48:          0          0          3          0   PCI-MSI-edge      eth5
>  49:          0          0          4         88   PCI-MSI-edge      eth6-rx-0
>  50:          0          0          0          0   PCI-MSI-edge      eth6-tx-0
>  51:          0          0          0          3   PCI-MSI-edge      eth6
>  52:          4          0          0         88   PCI-MSI-edge      eth7-rx-0
>  53:          0          0          0          0   PCI-MSI-edge      eth7-tx-0
>  54:          0          3          0          0   PCI-MSI-edge      eth7
> NMI:          0          0          0          0   Non-maskable interrupts
> LOC:     196962     205933     209322     208530   Local timer interrupts
> SPU:          0          0          0          0   Spurious interrupts
> PMI:          0          0          0          0   Performance
> monitoring interrupts
> PND:          0          0          0          0   Performance pending work
> RES:       4866        761        416        467   Rescheduling interrupts
> CAL:         39        102        107        109   Function call interrupts
> TLB:       6019       5481       4678       6759   TLB shootdowns
> TRM:          0          0          0          0   Thermal event interrupts
> THR:          0          0          0          0   Threshold APIC interrupts
> MCE:          0          0          0          0   Machine check exceptions
> MCP:          2          2          2          2   Machine check polls
> ERR:          3
> MIS:          0
> 
> 
> 
> > 3) ifconfig -a
> br0       Link encap:Ethernet  HWaddr 00:90:0b:14:ac:38
>           inet addr:192.168.164.222  Bcast:192.168.167.255  Mask:255.255.252.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:50 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:2300 (2.2 KiB)  TX bytes:0 (0.0 B)
> 
> eth0      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:38
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:10119 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:20075 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:6654384 (6.3 MiB)  TX bytes:1427890 (1.3 MiB)
>           Memory:fb580000-fb5a0000
> 
> eth1      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:39
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:20039 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:10155 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:1495356 (1.4 MiB)  TX bytes:6646438 (6.3 MiB)
>           Memory:fb5e0000-fb600000
> 
> eth2      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:3a
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:19190 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:9727 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:1432004 (1.3 MiB)  TX bytes:6365028 (6.0 MiB)
>           Memory:fb680000-fb6a0000
> 
> eth3      Link encap:Ethernet  HWaddr 00:90:0b:14:ac:3b
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:9689 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:19228 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:6372426 (6.0 MiB)  TX bytes:1367600 (1.3 MiB)
>           Memory:fb6e0000-fb700000
> eth4      Link encap:Ethernet  HWaddr 00:90:0b:15:79:18
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:9252 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:18354 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:6085260 (5.8 MiB)  TX bytes:1305576 (1.2 MiB)
>           Memory:fb8e0000-fb900000
> 
> eth5      Link encap:Ethernet  HWaddr 00:90:0b:15:79:19
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:18316 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:9290 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:1366920 (1.3 MiB)  TX bytes:6078300 (5.7 MiB)
>           Memory:fb9e0000-fba00000
> 
> eth6      Link encap:Ethernet  HWaddr 00:90:0b:15:79:1a
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:18709 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:9484 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:1396126 (1.3 MiB)  TX bytes:6204656 (5.9 MiB)
>           Memory:fbae0000-fbb00000
> 
> eth7      Link encap:Ethernet  HWaddr 00:90:0b:15:79:1b
>           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>           RX packets:9446 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:18746 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:6211812 (5.9 MiB)  TX bytes:1333280 (1.2 MiB)
>           Memory:fbbe0000-fbc00000
> 
> imq0      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           UP RUNNING NOARP  MTU:16000  Metric:1
>           RX packets:129112 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:129114 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>           RX bytes:31060964 (29.6 MiB)  TX bytes:31062207 (29.6 MiB)
> 
> imq1      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           NOARP  MTU:16000  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>  RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> imq2      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           NOARP  MTU:16000  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> imq3      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           NOARP  MTU:16000  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> imq4      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           NOARP  MTU:16000  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> imq5      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           NOARP  MTU:16000  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> imq6      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           NOARP  MTU:16000  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> imq7      Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>           NOARP  MTU:16000  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:11000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> 
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:14454 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:14454 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:1962561 (1.8 MiB)  TX bytes:1962561 (1.8 MiB)
> 
> 
> 
> > 4) cat /proc/sys/vm/min_free_kbytes
> 3806
> 
> > 5) cat /proc/meminfo
> 
> MemTotal:       12462064 kB
> MemFree:        12265628 kB
> Buffers:           29672 kB
> Cached:            64800 kB
> SwapCached:            0 kB
> Active:            87748 kB
> Inactive:          75756 kB
> Active(anon):      69040 kB
> Inactive(anon):      288 kB
> Active(file):      18708 kB
> Inactive(file):    75468 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> HighTotal:      11660872 kB
> HighFree:       11515792 kB
> LowTotal:         801192 kB
> LowFree:          749836 kB
> SwapTotal:      27226684 kB
> SwapFree:       27226684 kB
> Dirty:               120 kB
> Writeback:             0 kB
> AnonPages:         69032 kB
> Mapped:            14508 kB
> Shmem:               296 kB
> Slab:              15728 kB
> SReclaimable:       6044 kB
> SUnreclaim:         9684 kB
> KernelStack:        1376 kB
> PageTables:          996 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:    33457716 kB
> Committed_AS:     848336 kB
> VmallocTotal:     122880 kB
> VmallocUsed:        9468 kB
> VmallocChunk:     106708 kB
> DirectMap4k:        6136 kB
> DirectMap2M:      907264 kB
> 
> 
> > for each eth*
> >  ethtool -g eth*
> 
> e5570#ethtool -g eth0 - eth7 ( have the same config)
> Ring parameters for eth0:
> Pre-set maximums:
> RX:             4096
> RX Mini:        0
> RX Jumbo:       0
> TX:             4096
> Current hardware settings:
> RX:             256
> RX Mini:        0
> RX Jumbo:       0
> TX:             256

Thanks

You are running a 32bit kernel, so kernel can _not_ use more than 800
Mbytes of memory (Lowmem) for his own needs (including all network
buffers)

If you cannot switch to 64bit kernel, then you are forced to use lower
queue lengths (I see your imq devices use insane 11000 txqueuelen)

Each frame use 4K, maybe 16K, it depends on MTU.

even if we dont take into account other needs :
11000 * 16K = 170 Mbytes per imqX
1000 * 4K = 4Mbytes per ethX

170M * 8 -> memory overflow

Also /proc/sys/vm/min_free_kbytes  limit is probably too small.

Really, using a 32bit kernel on this machine is not wise.

Even reducing physical ram to 4GB would be better, you would get more
Lowmem memory ;)

Since vmalloc is not used very much on your machine you can reduce
vmalloc window from 128M to 32M to save 96MB of low mem
(boot parameter : vmalloc=32M )



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Kernel Panic on OOM with 10 HTB rules
  2010-09-08 14:45     ` Eric Dumazet
@ 2010-09-08 15:02       ` Krzysztof Olędzki
  2010-09-08 17:25       ` Jarek Poplawski
  1 sibling, 0 replies; 8+ messages in thread
From: Krzysztof Olędzki @ 2010-09-08 15:02 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Anand Raj Manickam, netdev, netfilter-devel, shemminger

On 2010-09-08 16:45, Eric Dumazet wrote:
<CUT>
> Thanks
>
> You are running a 32bit kernel, so kernel can _not_ use more than 800
> Mbytes of memory (Lowmem) for his own needs (including all network
> buffers)
>
> If you cannot switch to 64bit kernel, then you are forced to use lower
> queue lengths (I see your imq devices use insane 11000 txqueuelen)
>
> Each frame use 4K, maybe 16K, it depends on MTU.
>
> even if we dont take into account other needs :
> 11000 * 16K = 170 Mbytes per imqX
> 1000 * 4K = 4Mbytes per ethX
>
> 170M * 8 ->  memory overflow
>
> Also /proc/sys/vm/min_free_kbytes  limit is probably too small.
>
> Really, using a 32bit kernel on this machine is not wise.
>
> Even reducing physical ram to 4GB would be better, you would get more
> Lowmem memory ;)
>
> Since vmalloc is not used very much on your machine you can reduce
> vmalloc window from 128M to 32M to save 96MB of low mem
> (boot parameter : vmalloc=32M )

It is also possible to build a kernel using 2G/2G MEM SPLIT or even 
1G/3G if there is small userspace activity. But indeed, x86-64 is the 
proper solution to choose.

Best regards,

			Krzysztof Olędzki

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Kernel Panic on OOM with 10 HTB rules
  2010-09-08 14:45     ` Eric Dumazet
  2010-09-08 15:02       ` Krzysztof Olędzki
@ 2010-09-08 17:25       ` Jarek Poplawski
  1 sibling, 0 replies; 8+ messages in thread
From: Jarek Poplawski @ 2010-09-08 17:25 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Anand Raj Manickam, netdev, netfilter-devel, shemminger

Eric Dumazet wrote, On 09/08/2010 04:45 PM:

> Le mercredi 08 septembre 2010 à 19:39 +0530, Anand Raj Manickam a
> écrit :
>>
>> imq0      Link encap:UNSPEC  HWaddr
>> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>>           UP RUNNING NOARP  MTU:16000  Metric:1
>>           RX packets:129112 errors:0 dropped:0 overruns:0 frame:0
>>           TX packets:129114 errors:0 dropped:0 overruns:0 carrier:0
>>           collisions:0 txqueuelen:11000
>>           RX bytes:31060964 (29.6 MiB)  TX bytes:31062207 (29.6 MiB)
>>
> If you cannot switch to 64bit kernel, then you are forced to use lower
> queue lengths (I see your imq devices use insane 11000 txqueuelen)
> 
> Each frame use 4K, maybe 16K, it depends on MTU.
> 
> even if we dont take into account other needs :
> 11000 * 16K = 170 Mbytes per imqX
> 1000 * 4K = 4Mbytes per ethX
> 
> 170M * 8 -> memory overflow
> 

You should consider that htb creates by default one queue per class
with txqueuelen limit each. This should probably explain why your
problems start when you classify to many classes.

Jarek P.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Kernel Panic on OOM with 10 HTB rules
  2010-09-08 12:42 ` Eric Dumazet
  2010-09-08 14:09   ` Anand Raj Manickam
@ 2010-09-16  9:35   ` Patrick McHardy
  2010-09-16 12:20     ` Anand Raj Manickam
  1 sibling, 1 reply; 8+ messages in thread
From: Patrick McHardy @ 2010-09-16  9:35 UTC (permalink / raw)
  To: Anand Raj Manickam; +Cc: Eric Dumazet, netdev, netfilter-devel, shemminger

On 08.09.2010 14:42, Eric Dumazet wrote:
> Le mercredi 08 septembre 2010 à 16:22 +0530, Anand Raj Manickam a
> écrit :
>> I m struck with
>> Kernel Panic - Out of Memory Issue  when  traffic  is more than 100k
>> pps  and have  around 10 tc rules.
>>
>> There is NO kernel panic with 1 tc rule .
>>
>> I using a 2.6.34.6 ( latest stable kernel) with SMP enabled
>>
>> The rules are simple -
>> 10 iptable rules :
>> iptables -t mangle -I FORWARD -s x.x.x.x  -j CLASSIFY --set-class 0xz:0xy
>>
>> 10 tc rules :
>> tc  class add dev imq0 parent 1:z classid 1:y htb rate 2kbit ceil 32768kbps
>>
>> I m using  Intel Xeon Quad core x5570, 12GB RAM , with 8 e1000e , irq
>> affined ( 1 pair per core)
>>
>> There is NO issue / CRASH with multiple iptable rules marking to the
>> same tc rule/HTB class , i can sucessfully pass more than 200k pps /
>> 800Mbps traffic .
>>
>> But when each iptable rule is marking to seprate tc rule/HTB class and
>> when the traffic exceeds 100K pps / 20000 connections per second , we
>> hit this Out of Memory - Kernel panic condition .
>> Any help is appreciated . Do let me know if any more further
>> information is need.
> 
> Hi Anand
> 
> Could you give :
> 
> 1) A complete stack trace / panic report
> 2) cat /proc/interrupts
> 3) ifconfig -a
> 4) cat /proc/sys/vm/min_free_kbytes
> 5) cat /proc/meminfo
> 
> for each eth*
>  ethtool -g eth*

Please try to reproduce without the imq patch.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Kernel Panic on OOM with 10 HTB rules
  2010-09-16  9:35   ` Patrick McHardy
@ 2010-09-16 12:20     ` Anand Raj Manickam
  0 siblings, 0 replies; 8+ messages in thread
From: Anand Raj Manickam @ 2010-09-16 12:20 UTC (permalink / raw)
  To: Patrick McHardy; +Cc: Eric Dumazet, netdev, netfilter-devel

Things seem to be fine now after i reduced imq devs txqueuelen to 32
and MTU to 1600 and no of imqs to 2.
Thank you all for the Suggestions .
Anand

On Thu, Sep 16, 2010 at 3:05 PM, Patrick McHardy <kaber@trash.net> wrote:
> On 08.09.2010 14:42, Eric Dumazet wrote:
>> Le mercredi 08 septembre 2010 à 16:22 +0530, Anand Raj Manickam a
>> écrit :
>>> I m struck with
>>> Kernel Panic - Out of Memory Issue  when  traffic  is more than 100k
>>> pps  and have  around 10 tc rules.
>>>
>>> There is NO kernel panic with 1 tc rule .
>>>
>>> I using a 2.6.34.6 ( latest stable kernel) with SMP enabled
>>>
>>> The rules are simple -
>>> 10 iptable rules :
>>> iptables -t mangle -I FORWARD -s x.x.x.x  -j CLASSIFY --set-class 0xz:0xy
>>>
>>> 10 tc rules :
>>> tc  class add dev imq0 parent 1:z classid 1:y htb rate 2kbit ceil 32768kbps
>>>
>>> I m using  Intel Xeon Quad core x5570, 12GB RAM , with 8 e1000e , irq
>>> affined ( 1 pair per core)
>>>
>>> There is NO issue / CRASH with multiple iptable rules marking to the
>>> same tc rule/HTB class , i can sucessfully pass more than 200k pps /
>>> 800Mbps traffic .
>>>
>>> But when each iptable rule is marking to seprate tc rule/HTB class and
>>> when the traffic exceeds 100K pps / 20000 connections per second , we
>>> hit this Out of Memory - Kernel panic condition .
>>> Any help is appreciated . Do let me know if any more further
>>> information is need.
>>
>> Hi Anand
>>
>> Could you give :
>>
>> 1) A complete stack trace / panic report
>> 2) cat /proc/interrupts
>> 3) ifconfig -a
>> 4) cat /proc/sys/vm/min_free_kbytes
>> 5) cat /proc/meminfo
>>
>> for each eth*
>>  ethtool -g eth*
>
> Please try to reproduce without the imq patch.
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2010-09-16 12:20 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-09-08 10:52 Kernel Panic on OOM with 10 HTB rules Anand Raj Manickam
2010-09-08 12:42 ` Eric Dumazet
2010-09-08 14:09   ` Anand Raj Manickam
2010-09-08 14:45     ` Eric Dumazet
2010-09-08 15:02       ` Krzysztof Olędzki
2010-09-08 17:25       ` Jarek Poplawski
2010-09-16  9:35   ` Patrick McHardy
2010-09-16 12:20     ` Anand Raj Manickam

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.