[BUG] 2.2.19 -> 80% Packet Loss

[BUG] 2.2.19 -> 80% Packet Loss

[chuckw]

1. When pinging a machine using kernel 2.2.19 I consistently get an 80%
packet loss when doing a ping -f with a packet size of 64590 or higher.


2. A "ping -f -s 64589" to a machine running kernel 2.2.19 results in 0%
packet loss. By incrementing the packetsize by one "ping -f -s 64590"  or
higher, I consistently see 80% packet loss. ifconfig on the receiving
machine shows no anomolies.


3. 2.2.19, ping, flood, 64589, 64590, 80%, packet, loss


4. Linux version 2.2.19-7.0.1 (root@porky.devel.redhat.com) (gcc version
egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #1 Tue Apr 10 00:55:03
EDT 2001


5. There was no oops associated with this.


6. >>EOF
/bin/ping -f -s 64589
/bin/ping -f -s 64590
EOF


7.1  /usr/src/linux-2.2.19/scripts/ver_linux
Linux orchid 2.2.19-7.0.1 #1 Tue Apr 10 00:55:03 EDT 2001 i686 unknown

Gnu C                  2.96
Gnu make               3.79.1
binutils               2.10.0.18
util-linux             2.10r
modutils               2.3.21
e2fsprogs              1.18
Linux C Library        > libc.2.2
Dynamic linker (ldd)   2.2
Procps                 2.0.7
Net-tools              1.56
Console-tools          0.3.3
Sh-utils               2.0
Modules Loaded         autofs nfsd lockd sunrpc 3c59x agpgart usb-uhci
usbcore


7.2 cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 8
model name	: Pentium III (Coppermine)
stepping	: 3
cpu MHz		: 751.725
cache size	: 256 KB
fdiv_bug	: no
hlt_bug		: no
sep_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov
pat pse36 mmx fxsr xmm
bogomips	: 1500.77


7.3 cat /proc/modules
autofs                  9424   3 (autoclean)
nfsd                  182752   8 (autoclean)
lockd                  45264   1 (autoclean) [nfsd]
sunrpc                 61808   1 (autoclean) [nfsd lockd]
3c59x                  21584   1 (autoclean)
agpgart                18960   0 (unused)
usb-uhci               18736   0 (unused)
usbcore                43120   1 [usb-uhci]


7.4 No SCSI devices


7.5 /sbin/ifconfig (after sending it several pings with 80% packet loss)

eth0      Link encap:Ethernet  HWaddr 00:60:97:D7:60:E4
          inet addr:10.0.0.102  Bcast:10.0.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3972983 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5466442 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          Interrupt:12 Base address:0xa800

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:3924  Metric:1
          RX packets:210 errors:0 dropped:0 overruns:0 frame:0
          TX packets:210 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0


X. This has only been tested/observed on a 10Mb as well as a newer 100Mb
LAN. This has also been observed on machines running kernel 2.2.17. Non
Linux machines (NT and Win98) were tested in the same manner and do not
show the same symptoms. I reviewed the LKML archives, grep'd in the
sources and did a string on the kernel binary and was not able to find any
useful reference to 64590, 64589 or ping losses due to packet size.

-- 
Chuck Wolber
System Administrator
AltaServ Corporation
(425)576-1202
ten.vresatla@wkcuhc

Quidquid latine dictum sit, altum viditur.

Re: [BUG] 2.2.19 -> 80% Packet Loss

[Scott Laird]

Odds are it's a raw socket receive buffer issue.  Stock pings only ask for
a ~96k socket buffer, which means that they can only hold one ~64k packet
at a time.  So, if you're ever slow grabbing packets out of the buffer,
you're going to drop traffic.

You can fix this by upping the socket buffer that ping asks for (look for
setsockopt( ... SO_RCVBUF ...)) and then tuning the kernel to allow larger
socket buffers.  The file to fiddle with is /proc/sys/net/core/rmem_max.

That doesn't really answer why you'd want to fling that many 64k-ish ping
packets around, though.


Scott

On Thu, 14 Jun 2001 chuckw@altaserv.net wrote:

>
> 1. When pinging a machine using kernel 2.2.19 I consistently get an 80%
> packet loss when doing a ping -f with a packet size of 64590 or higher.
>
>
> 2. A "ping -f -s 64589" to a machine running kernel 2.2.19 results in 0%
> packet loss. By incrementing the packetsize by one "ping -f -s 64590"  or
> higher, I consistently see 80% packet loss. ifconfig on the receiving
> machine shows no anomolies.
>
>
> 3. 2.2.19, ping, flood, 64589, 64590, 80%, packet, loss
>
>
> 4. Linux version 2.2.19-7.0.1 (root@porky.devel.redhat.com) (gcc version
> egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #1 Tue Apr 10 00:55:03
> EDT 2001
>
>
> 5. There was no oops associated with this.
>
>
> 6. >>EOF
> /bin/ping -f -s 64589
> /bin/ping -f -s 64590
> EOF
>
>
> 7.1  /usr/src/linux-2.2.19/scripts/ver_linux
> Linux orchid 2.2.19-7.0.1 #1 Tue Apr 10 00:55:03 EDT 2001 i686 unknown
>
> Gnu C                  2.96
> Gnu make               3.79.1
> binutils               2.10.0.18
> util-linux             2.10r
> modutils               2.3.21
> e2fsprogs              1.18
> Linux C Library        > libc.2.2
> Dynamic linker (ldd)   2.2
> Procps                 2.0.7
> Net-tools              1.56
> Console-tools          0.3.3
> Sh-utils               2.0
> Modules Loaded         autofs nfsd lockd sunrpc 3c59x agpgart usb-uhci
> usbcore
>
>
> 7.2 cat /proc/cpuinfo
> processor	: 0
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 8
> model name	: Pentium III (Coppermine)
> stepping	: 3
> cpu MHz		: 751.725
> cache size	: 256 KB
> fdiv_bug	: no
> hlt_bug		: no
> sep_bug		: no
> f00f_bug	: no
> coma_bug	: no
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 2
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov
> pat pse36 mmx fxsr xmm
> bogomips	: 1500.77
>
>
> 7.3 cat /proc/modules
> autofs                  9424   3 (autoclean)
> nfsd                  182752   8 (autoclean)
> lockd                  45264   1 (autoclean) [nfsd]
> sunrpc                 61808   1 (autoclean) [nfsd lockd]
> 3c59x                  21584   1 (autoclean)
> agpgart                18960   0 (unused)
> usb-uhci               18736   0 (unused)
> usbcore                43120   1 [usb-uhci]
>
>
> 7.4 No SCSI devices
>
>
> 7.5 /sbin/ifconfig (after sending it several pings with 80% packet loss)
>
> eth0      Link encap:Ethernet  HWaddr 00:60:97:D7:60:E4
>           inet addr:10.0.0.102  Bcast:10.0.0.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:3972983 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5466442 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:100
>           Interrupt:12 Base address:0xa800
>
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           UP LOOPBACK RUNNING  MTU:3924  Metric:1
>           RX packets:210 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:210 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>
>
> X. This has only been tested/observed on a 10Mb as well as a newer 100Mb
> LAN. This has also been observed on machines running kernel 2.2.17. Non
> Linux machines (NT and Win98) were tested in the same manner and do not
> show the same symptoms. I reviewed the LKML archives, grep'd in the
> sources and did a string on the kernel binary and was not able to find any
> useful reference to 64590, 64589 or ping losses due to packet size.
>
> --
> Chuck Wolber
> System Administrator
> AltaServ Corporation
> (425)576-1202
> ten.vresatla@wkcuhc
>
> Quidquid latine dictum sit, altum viditur.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [BUG] 2.2.19 -> 80% Packet Loss

[José Luis Domingo López]

On Thursday, 14 June 2001, at 14:17:11 -0700,
chuckw@altaserv.net wrote:

> 
> 1. When pinging a machine using kernel 2.2.19 I consistently get an 80%
> packet loss when doing a ping -f with a packet size of 64590 or higher.
> 
What happens here is (under kernel 2.2.19):
ping -f -s 49092 localhost -->>   0 % packet loss
ping -f -s 49093 localhost -->> 100 % packet loss

Maybe this has something to do with fragmentation of IP packets to fit in
the underlying protocol's MTU (3929 in my loopback device).

When pinging from the network, it is expected to get an increasing number
of lost packets due to collisions in the medium. Moreover, IP
fragmentation, encapsulation and reassembly can be a reason for lost
packets, but is clear that what you get is quite strange :)

What I have noticed is that pinging from the network with large packet
sizes (for example, ping -s 55000 destinationIP), causes the ping program
to sometimes dump a echo_reply packet to stdout with the following message:

55008 bytes from 192.168.1.10: icmp_seq=8 ttl=255 time=145.9 ms
wrong data byte #190 should be 0xbe but was 0x3e

And some other times I get ping results with interesting patterns:

55008 bytes from 192.168.1.10: icmp_seq=45 ttl=255 time=1155.6 ms
55008 bytes from 192.168.1.10: icmp_seq=46 ttl=255 time=143.3 ms
55008 bytes from 192.168.1.10: icmp_seq=47 ttl=255 time=1155.2 ms
55008 bytes from 192.168.1.10: icmp_seq=48 ttl=255 time=143.8 ms


On a slower box, I get the following (kernel 2.4.4):
ping -f -s 32293 localhost -->>   0 % packet loss
ping -f -s 32294 localhost -->> 100 % packet loss

I've no idea what this could mean.

--
José Luis Domingo López
Linux Registered User #189436     Debian GNU/Linux Potato (P166 64 MB RAM)
 
jdomingo EN internautas PUNTO org  => ¿ Spam ? Atente a las consecuencias
jdomingo AT internautas DOT   org  => Spam at your own risk

Re: [BUG] 2.2.19 -> 80% Packet Loss

[Tim Moore]

chuckw@altaserv.net wrote:
> ...
> 2. A "ping -f -s 64589" to a machine running kernel 2.2.19 results in 0%
> packet loss. By incrementing the packetsize by one "ping -f -s 64590"  or
> higher, I consistently see 80% packet loss. ifconfig on the receiving
> machine shows no anomolies.
> ...
> 4. Linux version 2.2.19-7.0.1 (root@porky.devel.redhat.com) (gcc version
> egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)) #1 Tue Apr 10 00:55:03
> EDT 2001

No problems here.

rgds,
tim.

[note: -ai = athlon kernel + ide patch]

[tim@abit cron.daily]# cat /proc/version
Linux version 2.2.20p2-ai (root@abit) (gcc version egcs-2.91.66
19990314/Linux (egcs-1.1.2 release)) #12 Fri May 25 16:31:02 PDT 2001

[tim@abit cron.daily]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:A0:CC:57:89:93  
          inet addr:192.168.1.10  Bcast:192.168.1.255 
Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10204 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15280 errors:1 dropped:0 overruns:0 carrier:1
          collisions:0 txqueuelen:100 
          Interrupt:11 Base address:0xec00 

[tim@abit cron.daily]# ping -f -s 64590 dell
PING dell.yoyodyne.org (192.168.1.11) from 192.168.1.10 : 64590(64618)
bytes of data.
Warning: no SO_RCVTIMEO support, falling back to poll
..
--- dell.yoyodyne.org ping statistics ---
639 packets transmitted, 637 packets received, 0% packet loss
round-trip min/avg/max/mdev = 15.852/16.406/38.972/2.068 ms
[tim@abit cron.daily]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:A0:CC:57:89:93  
          inet addr:192.168.1.10  Bcast:192.168.1.255 
Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:38312 errors:0 dropped:0 overruns:0 frame:0
          TX packets:43421 errors:3 dropped:0 overruns:2 carrier:1
          collisions:0 txqueuelen:100 
          Interrupt:11 Base address:0xec00 

[tim@abit cron.daily]# ping dell -c 3
PING dell.yoyodyne.org (192.168.1.11) from 192.168.1.10 : 56(84) bytes
of data.
64 bytes from dell.yoyodyne.org (192.168.1.11): icmp_seq=0 ttl=255
time=334 usec
64 bytes from dell.yoyodyne.org (192.168.1.11): icmp_seq=1 ttl=255
time=294 usec
64 bytes from dell.yoyodyne.org (192.168.1.11): icmp_seq=2 ttl=255
time=296 usec

--- dell.yoyodyne.org ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/mdev = 0.294/0.308/0.334/0.018 ms


[tim@abit cron.daily]# ifconfig lo
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:3924  Metric:1
          RX packets:724500 errors:0 dropped:0 overruns:0 frame:0
          TX packets:724500 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 

[tim@abit cron.daily]# ping -f -s 64590 localhost
PING localhost (127.0.0.1) from 127.0.0.1 : 64590(64618) bytes of data.
Warning: no SO_RCVTIMEO support, falling back to poll
.
--- localhost ping statistics ---
7754 packets transmitted, 7754 packets received, 0% packet loss
round-trip min/avg/max/mdev = 0.960/1.011/25.556/0.623 ms
[tim@abit cron.daily]# ifconfig lo
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:3924  Metric:1
          RX packets:988136 errors:0 dropped:0 overruns:0 frame:0
          TX packets:988136 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 

--

Re: [BUG] 2.2.19 -> 80% Packet Loss

[Tim Moore]

José Luis Domingo López wrote:
> 
> On Thursday, 14 June 2001, at 14:17:11 -0700,
> chuckw@altaserv.net wrote:
> 
> >
> > 1. When pinging a machine using kernel 2.2.19 I consistently get an 80%
> > packet loss when doing a ping -f with a packet size of 64590 or higher.
> >
> What happens here is (under kernel 2.2.19):
> ping -f -s 49092 localhost -->>   0 % packet loss
> ping -f -s 49093 localhost -->> 100 % packet loss

[tim@abit cron.daily]# ping -w 10 -f -s 49093 localhost
PING localhost (127.0.0.1) from 127.0.0.1 : 49093(49121) bytes of data.
Warning: no SO_RCVTIMEO support, falling back to poll
.
--- localhost ping statistics ---
8051 packets transmitted, 8051 packets received, 0% packet loss
round-trip min/avg/max/mdev = 0.517/0.751/25.336/0.678 ms

> Maybe this has something to do with fragmentation of IP packets to fit in
> the underlying protocol's MTU (3929 in my loopback device).

[tim@abit cron.daily]# ifconfig lo
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:3924  Metric:1
          RX packets:1197462 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1197462 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 

[tim@abit cron.daily]# cat /proc/version
Linux version 2.2.20p2-ai (root@abit) (gcc version egcs-2.91.66
19990314/Linux (egcs-1.1.2 release)) #12 Fri May 25 16:31:02 PDT 2001

--

Re: [BUG] 2.2.19 -> 80% Packet Loss

[chuckw]

> You can fix this by upping the socket buffer that ping asks for (look
> for setsockopt( ... SO_RCVBUF ...)) and then tuning the kernel to
> allow larger socket buffers.  The file to fiddle with is
> /proc/sys/net/core/rmem_max.

Currently it is set to 65535. I doubled it several times and each time saw
no change when I sent it a ping flood with packet size 64590 or higher.
What sort of magnitude were you thinking?


> That doesn't really answer why you'd want to fling that many 64k-ish
> ping packets around, though.

No reason specifically other than intuition. Just kinda stumbled on to
this one.

-Chuck


-- 
Chuck Wolber
System Administrator
AltaServ Corporation
(425)576-1202
ten.vresatla@wkcuhc

Quidquid latine dictum sit, altum viditur.

Re: [BUG] 2.2.19 -> 80% Packet Loss

[Scott Laird]

On Fri, 15 Jun 2001 chuckw@altaserv.net wrote:
>
> > You can fix this by upping the socket buffer that ping asks for (look
> > for setsockopt( ... SO_RCVBUF ...)) and then tuning the kernel to
> > allow larger socket buffers.  The file to fiddle with is
> > /proc/sys/net/core/rmem_max.
>
> Currently it is set to 65535. I doubled it several times and each time saw
> no change when I sent it a ping flood with packet size 64590 or higher.
> What sort of magnitude were you thinking?

Did you change both /proc/sys/net/core/rmem_max *and* ping's setsockopt?
Do an strace on ping and see what's happening.


Scott