netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* tainted warnings with tcp splicing in 3.7.1
@ 2013-01-09 13:01 Christian Becker
  2013-01-09 14:50 ` Lukas Tribus
  2013-01-09 17:01 ` Eric Dumazet
  0 siblings, 2 replies; 17+ messages in thread
From: Christian Becker @ 2013-01-09 13:01 UTC (permalink / raw)
  To: netdev

Hi,

we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO.

Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy.

We´ve built the kernel yesterday from the 3.7.1 sources without any modifications.

The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs:

Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2)
Emulex Corporation OneConnect 10Gb NIC (Driver be2net)

The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic.

We´ve downgraded again to our old kernel version, but i guess you should take a look at this.

There are two kinds of messages:

an  9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------
Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
Jan  9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1
Jan  9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace:
Jan  9 11:34:28 srv11 kernel: [ 1081.573762]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:34:28 srv11 kernel: [ 1081.596750]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:34:28 srv11 kernel: [ 1081.617853]  [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe
Jan  9 11:34:28 srv11 kernel: [ 1081.639111]  [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc
Jan  9 11:34:28 srv11 kernel: [ 1081.661541]  [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194
Jan  9 11:34:28 srv11 kernel: [ 1081.681164]  [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b
Jan  9 11:34:28 srv11 kernel: [ 1081.703355]  [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223
Jan  9 11:34:28 srv11 kernel: [ 1081.724912]  [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0
Jan  9 11:34:28 srv11 kernel: [ 1081.744525]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]---

Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0
Jan  9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G        W    3.7.1 #1
Jan  9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace:
Jan  9 11:52:42 srv11 kernel: [ 2175.122727]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:52:42 srv11 kernel: [ 2175.147250]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:52:42 srv11 kernel: [ 2175.170289]  [<ffffffff81295f64>] ? release_sock+0xe6/0x11c
Jan  9 11:52:43 srv11 kernel: [ 2175.192639]  [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de
Jan  9 11:52:43 srv11 kernel: [ 2175.215020]  [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946
Jan  9 11:52:43 srv11 kernel: [ 2175.237682]  [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75
Jan  9 11:52:43 srv11 kernel: [ 2175.259727]  [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e
Jan  9 11:52:43 srv11 kernel: [ 2175.281111]  [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50
Jan  9 11:52:43 srv11 kernel: [ 2175.300573]  [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120
Jan  9 11:52:43 srv11 kernel: [ 2175.320073]  [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6
Jan  9 11:52:43 srv11 kernel: [ 2175.341151]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]---

As you can see here, the messages are appearing every couple of minutes:

Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()

^ permalink raw reply	[flat|nested] 17+ messages in thread
* RE: tainted warnings with tcp splicing in 3.7.1
  2013-01-09 13:01 tainted warnings with tcp splicing in 3.7.1 Christian Becker
@ 2013-01-09 14:50 ` Lukas Tribus
  2013-01-09 17:01 ` Eric Dumazet
  1 sibling, 0 replies; 17+ messages in thread
From: Lukas Tribus @ 2013-01-09 14:50 UTC (permalink / raw)
  To: netdev


For the record:
Another user reported similar warnings in a 3.5.0 kernel back in September [1] on the haproxy mailing list with the following kernel warnings:

[142654.793193] ------------[ cut here ]------------
[142654.793395] WARNING: at net/ipv4/tcp.c:1301 tcp_cleanup_rbuf+0x54/0x150()
[142654.793972] Hardware name: System Product Name
[142654.794573] cleanup rbuf bug: copied 68D1EF11 seq 68CFF65F rcvnxt 68D3565D
[142654.795165] Modules linked in: ixgbe(O) binfmt_misc 8021q fcoe
garp stp llc libfcoe libfc scsi_transport_fc scsi_tgt ip6t_REJECT nf
_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel
snd_hda_codec nouveau snd_hwdep igb snd_seq snd_seq_device snd_pcm
eeepc_wmi asus_wmi ttm drm_kms_helper sparse_keymap snd_timer snd drm
coretemp rfkill i2c_algo_bit mxm_wmi wmi video lpc_ich mfd_core
crc32c_intel r8169 i2c_i801 i2c_core soundcore snd_page_alloc mii mdio
pcspkr serio_raw ghash_clmulni_intel microcode uinput [last unloaded:
ixgbe]
[142654.798215] Pid: 18374, comm: haproxy Tainted: G        W  O 3.5.0 #1
[142654.798838] Call Trace:
[142654.799440]  [<ffffffff810422cf>] warn_slowpath_common+0x7f/0xc0
[142654.800031]  [<ffffffff810423c6>] warn_slowpath_fmt+0x46/0x50
[142654.800653]  [<ffffffff8147c270>] ? sock_pipe_buf_release+0x20/0x20
[142654.801237]  [<ffffffff814cf294>] tcp_cleanup_rbuf+0x54/0x150
[142654.801847]  [<ffffffff814d0ae1>] tcp_read_sock+0x1b1/0x200
[142654.802440]  [<ffffffff81472777>] ? sock_sendpage+0x27/0x30
[142654.803037]  [<ffffffff814ccd60>] ? tcp_done+0x90/0x90
[142654.803644]  [<ffffffff814d0bf0>] tcp_splice_read+0xc0/0x250
[142654.804239]  [<ffffffff814726b2>] sock_splice_read+0x62/0x80
[142654.804843]  [<ffffffff8118c73b>] do_splice_to+0x7b/0xa0
[142654.805457]  [<ffffffff8118e850>] sys_splice+0x540/0x560
[142654.806040]  [<ffffffff8159aed2>] system_call_fastpath+0x16/0x1b
[142654.806646] ---[ end trace 46d7fb693af33fde ]---



[1] http://permalink.gmane.org/gmane.comp.web.haproxy/9559 		 	   		  -

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-09 13:01 tainted warnings with tcp splicing in 3.7.1 Christian Becker
  2013-01-09 14:50 ` Lukas Tribus
@ 2013-01-09 17:01 ` Eric Dumazet
  2013-01-09 17:09   ` Eric Dumazet
  1 sibling, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2013-01-09 17:01 UTC (permalink / raw)
  To: Christian Becker; +Cc: netdev

On Wed, 2013-01-09 at 13:01 +0000, Christian Becker wrote:
> Hi,
> 
> we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO.
> 
> Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy.
> 
> We´ve built the kernel yesterday from the 3.7.1 sources without any modifications.
> 
> The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs:
> 
> Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2)
> Emulex Corporation OneConnect 10Gb NIC (Driver be2net)
> 
> The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic.
> 
> We´ve downgraded again to our old kernel version, but i guess you should take a look at this.
> 
> There are two kinds of messages:
> 
> an  9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------
> Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
> Jan  9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
> Jan  9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
> net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
> d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
> Jan  9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1
> Jan  9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace:
> Jan  9 11:34:28 srv11 kernel: [ 1081.573762]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
> Jan  9 11:34:28 srv11 kernel: [ 1081.596750]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
> Jan  9 11:34:28 srv11 kernel: [ 1081.617853]  [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe
> Jan  9 11:34:28 srv11 kernel: [ 1081.639111]  [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc
> Jan  9 11:34:28 srv11 kernel: [ 1081.661541]  [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194
> Jan  9 11:34:28 srv11 kernel: [ 1081.681164]  [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b
> Jan  9 11:34:28 srv11 kernel: [ 1081.703355]  [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223
> Jan  9 11:34:28 srv11 kernel: [ 1081.724912]  [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0
> Jan  9 11:34:28 srv11 kernel: [ 1081.744525]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
> Jan  9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]---
> 
> Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]-
> Jan  9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0
> Jan  9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
> net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
> d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
> Jan  9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G        W    3.7.1 #1
> Jan  9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace:
> Jan  9 11:52:42 srv11 kernel: [ 2175.122727]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
> Jan  9 11:52:42 srv11 kernel: [ 2175.147250]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
> Jan  9 11:52:42 srv11 kernel: [ 2175.170289]  [<ffffffff81295f64>] ? release_sock+0xe6/0x11c
> Jan  9 11:52:43 srv11 kernel: [ 2175.192639]  [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de
> Jan  9 11:52:43 srv11 kernel: [ 2175.215020]  [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946
> Jan  9 11:52:43 srv11 kernel: [ 2175.237682]  [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75
> Jan  9 11:52:43 srv11 kernel: [ 2175.259727]  [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e
> Jan  9 11:52:43 srv11 kernel: [ 2175.281111]  [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50
> Jan  9 11:52:43 srv11 kernel: [ 2175.300573]  [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120
> Jan  9 11:52:43 srv11 kernel: [ 2175.320073]  [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6
> Jan  9 11:52:43 srv11 kernel: [ 2175.341151]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
> Jan  9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]---
> 
> As you can see here, the messages are appearing every couple of minutes:
> 
> Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> Jan  9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()--

Thanks for this report

This might be a bug because of TCP collapsing

A "netstat -s" would  have been a very useful information.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-09 17:01 ` Eric Dumazet
@ 2013-01-09 17:09   ` Eric Dumazet
  2013-01-10  6:59     ` Eric Dumazet
  0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2013-01-09 17:09 UTC (permalink / raw)
  To: Christian Becker; +Cc: netdev, Willy Tarreau

On Wed, 2013-01-09 at 09:01 -0800, Eric Dumazet wrote:
> On Wed, 2013-01-09 at 13:01 +0000, Christian Becker wrote:
> > Hi,
> > 
> > we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO.
> > 
> > Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy.
> > 
> > We´ve built the kernel yesterday from the 3.7.1 sources without any modifications.
> > 
> > The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs:
> > 
> > Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2)
> > Emulex Corporation OneConnect 10Gb NIC (Driver be2net)
> > 
> > The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic.
> > 
> > We´ve downgraded again to our old kernel version, but i guess you should take a look at this.
> > 
> > There are two kinds of messages:
> > 
> > an  9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------
> > Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
> > Jan  9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
> > Jan  9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
> > net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
> > d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
> > Jan  9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1
> > Jan  9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace:
> > Jan  9 11:34:28 srv11 kernel: [ 1081.573762]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
> > Jan  9 11:34:28 srv11 kernel: [ 1081.596750]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
> > Jan  9 11:34:28 srv11 kernel: [ 1081.617853]  [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe
> > Jan  9 11:34:28 srv11 kernel: [ 1081.639111]  [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc
> > Jan  9 11:34:28 srv11 kernel: [ 1081.661541]  [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194
> > Jan  9 11:34:28 srv11 kernel: [ 1081.681164]  [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b
> > Jan  9 11:34:28 srv11 kernel: [ 1081.703355]  [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223
> > Jan  9 11:34:28 srv11 kernel: [ 1081.724912]  [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0
> > Jan  9 11:34:28 srv11 kernel: [ 1081.744525]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
> > Jan  9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]---
> > 
> > Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]-
> > Jan  9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0
> > Jan  9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
> > net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
> > d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
> > Jan  9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G        W    3.7.1 #1
> > Jan  9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace:
> > Jan  9 11:52:42 srv11 kernel: [ 2175.122727]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
> > Jan  9 11:52:42 srv11 kernel: [ 2175.147250]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
> > Jan  9 11:52:42 srv11 kernel: [ 2175.170289]  [<ffffffff81295f64>] ? release_sock+0xe6/0x11c
> > Jan  9 11:52:43 srv11 kernel: [ 2175.192639]  [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de
> > Jan  9 11:52:43 srv11 kernel: [ 2175.215020]  [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946
> > Jan  9 11:52:43 srv11 kernel: [ 2175.237682]  [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75
> > Jan  9 11:52:43 srv11 kernel: [ 2175.259727]  [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e
> > Jan  9 11:52:43 srv11 kernel: [ 2175.281111]  [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50
> > Jan  9 11:52:43 srv11 kernel: [ 2175.300573]  [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120
> > Jan  9 11:52:43 srv11 kernel: [ 2175.320073]  [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6
> > Jan  9 11:52:43 srv11 kernel: [ 2175.341151]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
> > Jan  9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]---
> > 
> > As you can see here, the messages are appearing every couple of minutes:
> > 
> > Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> > Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
> > Jan  9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()--
> 
> Thanks for this report
> 
> This might be a bug because of TCP collapsing
> 
> A "netstat -s" would  have been a very useful information.
> 
> 

My feeling is that tcp_recv_skb() should eat skbs instead of only
finding the right one

Thats because skb_splice_bits() releases the socket lock before calling
splice_to_pipe()

Once socket is released, other incoming TCP frames can be processed, and
the skb we are actually processing might be 'collapsed' into smaller
units.

Christian, if I send you patches, are you OK to test them ?

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-09 17:09   ` Eric Dumazet
@ 2013-01-10  6:59     ` Eric Dumazet
  2013-01-10  7:21       ` Willy Tarreau
                         ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Eric Dumazet @ 2013-01-10  6:59 UTC (permalink / raw)
  To: Christian Becker, David Miller; +Cc: netdev, Willy Tarreau

From: Eric Dumazet <edumazet@google.com>

On Wed, 2013-01-09 at 09:09 -0800, Eric Dumazet wrote:

> My feeling is that tcp_recv_skb() should eat skbs instead of only
> finding the right one
> 

Thats indeed the case.

> Thats because skb_splice_bits() releases the socket lock before calling
> splice_to_pipe()
> 
> Once socket is released, other incoming TCP frames can be processed, and
> the skb we are actually processing might be 'collapsed' into smaller
> units.
> 
> Christian, if I send you patches, are you OK to test them ?
> 
> 

Here is the patch fixing this issue.

Thanks a lot for your report, this is a very very old bug.

GRO being more deployed, and with TCP coalescing as well, chances to
trigger this bug increased a lot.

To reproduce it, I had to force MSS=400 and stress the receiver,
adding extra delays in skb_splice_bits() with socket lock being not
held.



[PATCH] tcp: fix splice() and tcp collapsing interaction

Under unusual circumstances, TCP collapse can split a big GRO TCP packet
while its being used in a splice(socket->pipe) operation.

skb_splice_bits() releases the socket lock before calling
splice_to_pipe().

[ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
[ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
[ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13

To fix this problem, we must eat skbs in tcp_recv_skb().

Remove the inline keyword from tcp_recv_skb() definition since
it has three call sites.

Reported-by: Christian Becker <c.becker@traviangames.com>
Cc: Willy Tarreau <w@1wt.eu>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 net/ipv4/tcp.c |   13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 1ca2536..1f901be 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1428,12 +1428,12 @@ static void tcp_service_net_dma(struct sock *sk, bool wait)
 }
 #endif
 
-static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
+static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 {
 	struct sk_buff *skb;
 	u32 offset;
 
-	skb_queue_walk(&sk->sk_receive_queue, skb) {
+	while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
 		offset = seq - TCP_SKB_CB(skb)->seq;
 		if (tcp_hdr(skb)->syn)
 			offset--;
@@ -1441,6 +1441,11 @@ static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 			*off = offset;
 			return skb;
 		}
+		/* This looks weird, but this can happen if TCP collapsing
+		 * splitted a fat GRO packet, while we released socket lock
+		 * in skb_splice_bits()
+		 */
+		sk_eat_skb(sk, skb, false);
 	}
 	return NULL;
 }
@@ -1520,8 +1525,10 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc,
 	tcp_rcv_space_adjust(sk);
 
 	/* Clean up data we have read: This will do ACK frames. */
-	if (copied > 0)
+	if (copied > 0) {
+		tcp_recv_skb(sk, seq, &offset);
 		tcp_cleanup_rbuf(sk, copied);
+	}
 	return copied;
 }
 EXPORT_SYMBOL(tcp_read_sock);

^ permalink raw reply related	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10  6:59     ` Eric Dumazet
@ 2013-01-10  7:21       ` Willy Tarreau
  2013-01-10 15:29         ` Eric Dumazet
  2013-01-10 18:27       ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus
  2013-01-10 22:39       ` David Miller
  2 siblings, 1 reply; 17+ messages in thread
From: Willy Tarreau @ 2013-01-10  7:21 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Christian Becker, David Miller, netdev

On Wed, Jan 09, 2013 at 10:59:09PM -0800, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> On Wed, 2013-01-09 at 09:09 -0800, Eric Dumazet wrote:
> 
> > My feeling is that tcp_recv_skb() should eat skbs instead of only
> > finding the right one
> > 
> 
> Thats indeed the case.
> 
> > Thats because skb_splice_bits() releases the socket lock before calling
> > splice_to_pipe()
> > 
> > Once socket is released, other incoming TCP frames can be processed, and
> > the skb we are actually processing might be 'collapsed' into smaller
> > units.
> > 
> > Christian, if I send you patches, are you OK to test them ?
> > 
> > 
> 
> Here is the patch fixing this issue.
> 
> Thanks a lot for your report, this is a very very old bug.
> 
> GRO being more deployed, and with TCP coalescing as well, chances to
> trigger this bug increased a lot.
> 
> To reproduce it, I had to force MSS=400 and stress the receiver,
> adding extra delays in skb_splice_bits() with socket lock being not
> held.

FWIW, I tested your patch here and did not notice any regression
compared to last week-end tests, at various MTU size combinations.

Cheers,
Willy

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10  7:21       ` Willy Tarreau
@ 2013-01-10 15:29         ` Eric Dumazet
  2013-01-10 16:20           ` Eric Dumazet
  2013-01-12  0:46           ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet
  0 siblings, 2 replies; 17+ messages in thread
From: Eric Dumazet @ 2013-01-10 15:29 UTC (permalink / raw)
  To: Willy Tarreau; +Cc: Christian Becker, David Miller, netdev

On Thu, 2013-01-10 at 08:21 +0100, Willy Tarreau wrote:

> FWIW, I tested your patch here and did not notice any regression
> compared to last week-end tests, at various MTU size combinations.
> 

Thanks Willy

Tested-by: Willy Tarreau <w@1wt.eu>

I believe I need to fix net-next, 

commit 9ca1b22d6d228177e6f929f6818a1cd3d5e30c4a
(net: splice: avoid high order page splitting)

missed the loopback case : the skb->head might need several
linear_to_page() calls.

I'll send a patch after full testing.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10 15:29         ` Eric Dumazet
@ 2013-01-10 16:20           ` Eric Dumazet
  2013-01-10 18:22             ` Rick Jones
  2013-01-12  0:46           ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet
  1 sibling, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2013-01-10 16:20 UTC (permalink / raw)
  To: Willy Tarreau, Rick Jones; +Cc: Christian Becker, David Miller, netdev

On Thu, 2013-01-10 at 07:29 -0800, Eric Dumazet wrote:
> On Thu, 2013-01-10 at 08:21 +0100, Willy Tarreau wrote:
> 
> > FWIW, I tested your patch here and did not notice any regression
> > compared to last week-end tests, at various MTU size combinations.
> > 
> 
> Thanks Willy

I also want to thanks Rick, as the latest netperf has splice() support.

Thanks Rick !

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10 16:20           ` Eric Dumazet
@ 2013-01-10 18:22             ` Rick Jones
  2013-01-10 18:42               ` Eric Dumazet
  0 siblings, 1 reply; 17+ messages in thread
From: Rick Jones @ 2013-01-10 18:22 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Willy Tarreau, Christian Becker, David Miller, netdev

On 01/10/2013 08:20 AM, Eric Dumazet wrote:
> I also want to thanks Rick, as the latest netperf has splice() support.
>
> Thanks Rick !

You are quite welcome - and thank you for helping me get it to actually 
work :)

Those wishing to try it themselves should grab the top-of-trunk netperf 
bits from http://www.netperf.org/svn/netperf2/trunk .  The use of 
splice() is gated by a test-specific -V option:

raj@tardy:~/netperf2_trunk/src$ ./netperf -t omni -- -d recv -V
OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 
localhost.localdomain () port 0 AF_INET : copy avoidance : demo
Remote      Local       Remote Elapsed Throughput Throughput
Send Socket Recv Socket Send   Time               Units
Size        Size        Size   (sec)
Final       Final
1661688     4194304     16384  10.00   26103.14   10^6bits/s

You should see that "copy avoidance" appearing in the test banner.  It 
will also "take" for things like a migrated TCP_mumble test.  For those 
cases where you don't see a throughput change, enabling CPU utilization 
measurement and looking at that and service demand should show a difference.

happy benchmarking,

rick jones

^ permalink raw reply	[flat|nested] 17+ messages in thread
* RE: tainted warnings with tcp splicing in 3.7.1
  2013-01-10  6:59     ` Eric Dumazet
  2013-01-10  7:21       ` Willy Tarreau
@ 2013-01-10 18:27       ` Lukas Tribus
  2013-01-10 18:37         ` Eric Dumazet
  2013-01-10 22:39       ` David Miller
  2 siblings, 1 reply; 17+ messages in thread
From: Lukas Tribus @ 2013-01-10 18:27 UTC (permalink / raw)
  To: eric.dumazet; +Cc: netdev


Hi Eric,

this is probably a dumb question ... but since the fix is in net/ipv4/tcp.c I was asking myself if this can affect IPv6 as well?



Thanks,

Lukas


> [PATCH] tcp: fix splice() and tcp collapsing interaction
> 
> Under unusual circumstances, TCP collapse can split a big GRO TCP packet
> while its being used in a splice(socket->pipe) operation.
> 
> skb_splice_bits() releases the socket lock before calling
> splice_to_pipe().
> 
> [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
> [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
> 
> To fix this problem, we must eat skbs in tcp_recv_skb().
> 
> Remove the inline keyword from tcp_recv_skb() definition since
> it has three call sites.
> 
> Reported-by: Christian Becker <c.becker@traviangames.com>
> Cc: Willy Tarreau <w@1wt.eu>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> ---
>  net/ipv4/tcp.c |   13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
> index 1ca2536..1f901be 100644
> --- a/net/ipv4/tcp.c
> +++ b/net/ipv4/tcp.c
> @@ -1428,12 +1428,12 @@ static void tcp_service_net_dma(struct sock *sk, bool wait)
>  }
>  #endif
>  
> -static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
> +static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
>  {
>  	struct sk_buff *skb;
>  	u32 offset;
>  
> -	skb_queue_walk(&sk->sk_receive_queue, skb) {
> +	while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
>  		offset = seq - TCP_SKB_CB(skb)->seq;
>  		if (tcp_hdr(skb)->syn)
>  			offset--;
> @@ -1441,6 +1441,11 @@ static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
>  			*off = offset;
>  			return skb;
>  		}
> +		/* This looks weird, but this can happen if TCP collapsing
> +		 * splitted a fat GRO packet, while we released socket lock
> +		 * in skb_splice_bits()
> +		 */
> +		sk_eat_skb(sk, skb, false);
>  	}
>  	return NULL;
>  }
> @@ -1520,8 +1525,10 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc,
>  	tcp_rcv_space_adjust(sk);
>  
>  	/* Clean up data we have read: This will do ACK frames. */
> -	if (copied > 0)
> +	if (copied > 0) {
> +		tcp_recv_skb(sk, seq, &offset);
>  		tcp_cleanup_rbuf(sk, copied);
> +	}
>  	return copied;
>  }
>  EXPORT_SYMBOL(tcp_read_sock);
> 
 		 	   		  

^ permalink raw reply	[flat|nested] 17+ messages in thread
* RE: tainted warnings with tcp splicing in 3.7.1
  2013-01-10 18:27       ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus
@ 2013-01-10 18:37         ` Eric Dumazet
  0 siblings, 0 replies; 17+ messages in thread
From: Eric Dumazet @ 2013-01-10 18:37 UTC (permalink / raw)
  To: Lukas Tribus; +Cc: netdev

On Thu, 2013-01-10 at 19:27 +0100, Lukas Tribus wrote:
> Hi Eric,
> 
> this is probably a dumb question ... but since the fix is in net/ipv4/tcp.c I was asking myself if this can affect IPv6 as well?
> 

Yes it can, net/ipv4 contains generic TCP code.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10 18:22             ` Rick Jones
@ 2013-01-10 18:42               ` Eric Dumazet
  2013-01-10 18:49                 ` Rick Jones
  0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2013-01-10 18:42 UTC (permalink / raw)
  To: Rick Jones; +Cc: Willy Tarreau, Christian Becker, David Miller, netdev

On Thu, 2013-01-10 at 10:22 -0800, Rick Jones wrote:
> On 01/10/2013 08:20 AM, Eric Dumazet wrote:
> > I also want to thanks Rick, as the latest netperf has splice() support.
> >
> > Thanks Rick !
> 
> You are quite welcome - and thank you for helping me get it to actually 
> work :)
> 
> Those wishing to try it themselves should grab the top-of-trunk netperf 
> bits from http://www.netperf.org/svn/netperf2/trunk .  The use of 
> splice() is gated by a test-specific -V option:
> 
> raj@tardy:~/netperf2_trunk/src$ ./netperf -t omni -- -d recv -V
> OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 
> localhost.localdomain () port 0 AF_INET : copy avoidance : demo
> Remote      Local       Remote Elapsed Throughput Throughput
> Send Socket Recv Socket Send   Time               Units
> Size        Size        Size   (sec)
> Final       Final
> 1661688     4194304     16384  10.00   26103.14   10^6bits/s
> 
> You should see that "copy avoidance" appearing in the test banner.  It 
> will also "take" for things like a migrated TCP_mumble test.  For those 
> cases where you don't see a throughput change, enabling CPU utilization 
> measurement and looking at that and service demand should show a difference.

Thanks Rick !

Can we use zero copy for the sender as well (sendfile() or vmsplice()) ?

Here are some results :

Reference time (no splice())

# ./netperf -H 127.0.0.1 -t omni -- -d recv     
OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 ()
port 0 AF_INET
catcher: timer popped with times_up != 0
Remote      Local       Remote Elapsed Throughput Throughput  
Send Socket Recv Socket Send   Time               Units       
Size        Size        Size   (sec)                          
Final       Final                                             
2097152     2097152     16384  10.00   33237.74   10^6bits/s  


zero copy at receiver (splice(socket ->pipe), splice(pipe -> /dev/null))

# ./netperf -H 127.0.0.1 -t omni -- -d recv -V
OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 ()
port 0 AF_INET : copy avoidance
catcher: timer popped with times_up != 0
Remote      Local       Remote Elapsed Throughput Throughput  
Send Socket Recv Socket Send   Time               Units       
Size        Size        Size   (sec)                          
Final       Final                                             
1325580     2097152     16384  10.00   51980.60   10^6bits/s  

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10 18:42               ` Eric Dumazet
@ 2013-01-10 18:49                 ` Rick Jones
  2013-01-10 19:43                   ` Willy Tarreau
  0 siblings, 1 reply; 17+ messages in thread
From: Rick Jones @ 2013-01-10 18:49 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Willy Tarreau, Christian Becker, David Miller, netdev

On 01/10/2013 10:42 AM, Eric Dumazet wrote:
> Can we use zero copy for the sender as well (sendfile() or vmsplice()) ?

Not at present.  The TCP_SENDFILE test has not been "migrated" and there 
is no corresponding sendfile() test in the omni code, so the attempt to 
enable copy-avoidance won't "take."

I'll take that as a feature request for netperf.

rick

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10 18:49                 ` Rick Jones
@ 2013-01-10 19:43                   ` Willy Tarreau
  0 siblings, 0 replies; 17+ messages in thread
From: Willy Tarreau @ 2013-01-10 19:43 UTC (permalink / raw)
  To: Rick Jones; +Cc: Eric Dumazet, Christian Becker, David Miller, netdev

On Thu, Jan 10, 2013 at 10:49:17AM -0800, Rick Jones wrote:
> On 01/10/2013 10:42 AM, Eric Dumazet wrote:
> >Can we use zero copy for the sender as well (sendfile() or vmsplice()) ?
> 
> Not at present.  The TCP_SENDFILE test has not been "migrated" and there 
> is no corresponding sendfile() test in the omni code, so the attempt to 
> enable copy-avoidance won't "take."

Note that in my httpterm server, I'm using tee+splice(). I had a problem
with vmsplice() in the past, I don't remember which one.

For the "inject" http client, I'm using recv(MSG_TRUNC) which is always
faster than splice() and does the job well, considering that the goal is
to count and destroy data very fast.

By combining them both on the same machine over the loopback, I see
70 Gbps on a core-2 duo 2.66 GHz. These are obviously not real gigs,
but at least it shows what the network stack can do when we remove
memory copies !

Regards,
Willy

^ permalink raw reply	[flat|nested] 17+ messages in thread
* Re: tainted warnings with tcp splicing in 3.7.1
  2013-01-10  6:59     ` Eric Dumazet
  2013-01-10  7:21       ` Willy Tarreau
  2013-01-10 18:27       ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus
@ 2013-01-10 22:39       ` David Miller
  2 siblings, 0 replies; 17+ messages in thread
From: David Miller @ 2013-01-10 22:39 UTC (permalink / raw)
  To: eric.dumazet; +Cc: c.becker, netdev, w

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Wed, 09 Jan 2013 22:59:09 -0800

> [PATCH] tcp: fix splice() and tcp collapsing interaction
> 
> Under unusual circumstances, TCP collapse can split a big GRO TCP packet
> while its being used in a splice(socket->pipe) operation.
> 
> skb_splice_bits() releases the socket lock before calling
> splice_to_pipe().
> 
> [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
> [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
> [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
> 
> To fix this problem, we must eat skbs in tcp_recv_skb().
> 
> Remove the inline keyword from tcp_recv_skb() definition since
> it has three call sites.
> 
> Reported-by: Christian Becker <c.becker@traviangames.com>
> Cc: Willy Tarreau <w@1wt.eu>
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Applied.

^ permalink raw reply	[flat|nested] 17+ messages in thread
* [PATCH net-next] net: splice: fix __splice_segment()
  2013-01-10 15:29         ` Eric Dumazet
  2013-01-10 16:20           ` Eric Dumazet
@ 2013-01-12  0:46           ` Eric Dumazet
  2013-01-12  0:48             ` David Miller
  1 sibling, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2013-01-12  0:46 UTC (permalink / raw)
  To: Willy Tarreau, David Miller; +Cc: netdev

From: Eric Dumazet <edumazet@google.com>

commit 9ca1b22d6d2 (net: splice: avoid high order page splitting)
forgot that skb->head could need a copy into several page frags.

This could be the case for loopback traffic mostly.

Also remove now useless skb argument from linear_to_page()
and __splice_segment() prototypes.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Willy Tarreau <w@1wt.eu>
---
 net/core/skbuff.c |   28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 1e1b9ea..2568c44 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1652,7 +1652,7 @@ static void sock_spd_release(struct splice_pipe_desc *spd, unsigned int i)
 
 static struct page *linear_to_page(struct page *page, unsigned int *len,
 				   unsigned int *offset,
-				   struct sk_buff *skb, struct sock *sk)
+				   struct sock *sk)
 {
 	struct page_frag *pfrag = sk_page_frag(sk);
 
@@ -1685,14 +1685,14 @@ static bool spd_can_coalesce(const struct splice_pipe_desc *spd,
 static bool spd_fill_page(struct splice_pipe_desc *spd,
 			  struct pipe_inode_info *pipe, struct page *page,
 			  unsigned int *len, unsigned int offset,
-			  struct sk_buff *skb, bool linear,
+			  bool linear,
 			  struct sock *sk)
 {
 	if (unlikely(spd->nr_pages == MAX_SKB_FRAGS))
 		return true;
 
 	if (linear) {
-		page = linear_to_page(page, len, &offset, skb, sk);
+		page = linear_to_page(page, len, &offset, sk);
 		if (!page)
 			return true;
 	}
@@ -1711,13 +1711,11 @@ static bool spd_fill_page(struct splice_pipe_desc *spd,
 
 static bool __splice_segment(struct page *page, unsigned int poff,
 			     unsigned int plen, unsigned int *off,
-			     unsigned int *len, struct sk_buff *skb,
+			     unsigned int *len,
 			     struct splice_pipe_desc *spd, bool linear,
 			     struct sock *sk,
 			     struct pipe_inode_info *pipe)
 {
-	unsigned int flen;
-
 	if (!*len)
 		return true;
 
@@ -1732,12 +1730,16 @@ static bool __splice_segment(struct page *page, unsigned int poff,
 	plen -= *off;
 	*off = 0;
 
-	flen = min(*len, plen);
-
-	if (spd_fill_page(spd, pipe, page, &flen, poff, skb, linear, sk))
-		return true;
+	do {
+		unsigned int flen = min(*len, plen);
 
-	*len -= flen;
+		if (spd_fill_page(spd, pipe, page, &flen, poff,
+				  linear, sk))
+			return true;
+		poff += flen;
+		plen -= flen;
+		*len -= flen;
+	} while (*len && plen);
 
 	return false;
 }
@@ -1760,7 +1762,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe,
 	if (__splice_segment(virt_to_page(skb->data),
 			     (unsigned long) skb->data & (PAGE_SIZE - 1),
 			     skb_headlen(skb),
-			     offset, len, skb, spd,
+			     offset, len, spd,
 			     skb_head_is_locked(skb),
 			     sk, pipe))
 		return true;
@@ -1773,7 +1775,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe,
 
 		if (__splice_segment(skb_frag_page(f),
 				     f->page_offset, skb_frag_size(f),
-				     offset, len, skb, spd, false, sk, pipe))
+				     offset, len, spd, false, sk, pipe))
 			return true;
 	}
 

^ permalink raw reply related	[flat|nested] 17+ messages in thread
* Re: [PATCH net-next] net: splice: fix __splice_segment()
  2013-01-12  0:46           ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet
@ 2013-01-12  0:48             ` David Miller
  0 siblings, 0 replies; 17+ messages in thread
From: David Miller @ 2013-01-12  0:48 UTC (permalink / raw)
  To: eric.dumazet; +Cc: w, netdev

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Fri, 11 Jan 2013 16:46:37 -0800

> From: Eric Dumazet <edumazet@google.com>
> 
> commit 9ca1b22d6d2 (net: splice: avoid high order page splitting)
> forgot that skb->head could need a copy into several page frags.
> 
> This could be the case for loopback traffic mostly.
> 
> Also remove now useless skb argument from linear_to_page()
> and __splice_segment() prototypes.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Willy Tarreau <w@1wt.eu>

Applied, thanks Eric.

^ permalink raw reply	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2013-01-12  0:48 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-01-09 13:01 tainted warnings with tcp splicing in 3.7.1 Christian Becker
2013-01-09 14:50 ` Lukas Tribus
2013-01-09 17:01 ` Eric Dumazet
2013-01-09 17:09   ` Eric Dumazet
2013-01-10  6:59     ` Eric Dumazet
2013-01-10  7:21       ` Willy Tarreau
2013-01-10 15:29         ` Eric Dumazet
2013-01-10 16:20           ` Eric Dumazet
2013-01-10 18:22             ` Rick Jones
2013-01-10 18:42               ` Eric Dumazet
2013-01-10 18:49                 ` Rick Jones
2013-01-10 19:43                   ` Willy Tarreau
2013-01-12  0:46           ` [PATCH net-next] net: splice: fix __splice_segment() Eric Dumazet
2013-01-12  0:48             ` David Miller
2013-01-10 18:27       ` tainted warnings with tcp splicing in 3.7.1 Lukas Tribus
2013-01-10 18:37         ` Eric Dumazet
2013-01-10 22:39       ` David Miller

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).