[Bug 60518] New: Heavy network traffic between guest and host triggers kernel oops

[Bug 60518] New: Heavy network traffic between guest and host triggers kernel oops

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=60518

            Bug ID: 60518
           Summary: Heavy network traffic between guest and host triggers
                    kernel oops
           Product: Virtualization
           Version: unspecified
    Kernel Version: v3.9.8 and v3.9.9
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: bvanassche@acm.org
        Regression: No

When sending network packets at a high rate between KVM guests I either run
into bug 60505 or into the call trace below:

BUG: unable to handle kernel NULL pointer dereference at 000000000000001c
IP: [<ffffffff810f73a9>] put_compound_page+0x89/0x170
PGD 0 
Oops: 0000 [#1] SMP 
Modules linked in: dm_queue_length dm_multipath rdma_ucm rdma_cm iw_cm ib_addr
ib_srp scsi_transport_srp scsi_tgt ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en
mlx4_ib ib_sa ib_mad ib_core mlx4_core libcrc32c crc32c_intel nls_utf8
iscsi_tcp libisc rt_iscsi vhost_net tun fuse ip6table_filter ip6_tables
iptable_filter ip_tables ebtable_nat ebtables x_tables af_packet bridge stp llc
dm_mod hid_generic usbhid hid snd_hda_codec_hdmi snd_hda_codec_realtek
snd_hda_intel snd_hda_codec acpi_cpuf vm_intel snd_pcm cdrom pcspkr kvm snd_seq
sg snd_timer snd_seq_device snd r8169 ehci_pci ehci_hcd i2c_i801 soundcore wmi
snd_page_alloc mii microcode autofs4 ext4 jbd2 mbcache crc16 raid456
async_raid6_recov async_pq raid6_pq async_xor xor asy id0 raid1 sd_mod
crc_t10dif ahci xhci_hcd libahci i915 drm_kms_helper drm intel_agp i2c_algo_bit
intel_gtt agpgart usbcore i2c_core video usb_common button processor
thermal_sys hwmon scsi_dh_alua scsi_dh pata_acpi libata scsi_mod [last unload 
CPU 3 
Pid: 30508, comm: vhost-30506 Tainted: G           O 3.9.8+ #1 Gigabyte
Technology Co., Ltd. Z68X-UD3H-B3/Z68X-UD3H-B3
RIP: 0010:[<ffffffff810f73a9>]  [<ffffffff810f73a9>]
put_compound_page+0x89/0x170
RSP: 0018:ffff880036887bc8  EFLAGS: 00010286
RAX: ffff8800cdb8ae00 RBX: ffff8800cdb8a800 RCX: ffff8800cdb8b6c0
RDX: 0000000000000140 RSI: 0000000000000001 RDI: ffff8800cdb8a800
RBP: ffff880036887be8 R08: ffff88005ddb4518 R09: 0000000000000010
R10: 0000000000000000 R11: 00007f6f00000000 R12: 0000000000000000
R13: ffffffffa079896c R14: 00000000000091aa R15: ffff880115200900
FS:  0000000000000000(0000) GS:ffff88011fac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000000001c CR3: 000000009f2ff000 CR4: 00000000000427e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-30506 (pid: 30508, threadinfo ffff880036886000, task
ffff8801088cc020)
Stack:
ffffea00016fbfc0 0000000000000012 ffff880115200900 ffffffffa079896c
ffff880036887bf8 ffffffff810f77ec ffff880036887c18 ffffffff8132046f
ffff880115200900 ffff880115200900 ffff880036887c38 ffffffff8132050e
Call Trace:
[<ffffffff810f77ec>] put_page+0x2c/0x40
[<ffffffff8132046f>] skb_release_data+0x8f/0x110
[<ffffffff8132050e>] __kfree_skb+0x1e/0xa0
[<ffffffff813205c6>] kfree_skb+0x36/0xa0
[<ffffffffa079896c>] tun_get_user+0x71c/0x810 [tun]
[<ffffffffa0798aba>] tun_sendmsg+0x5a/0x80 [tun]
[<ffffffffa07a75d7>] handle_tx+0x287/0x680 [vhost_net]
[<ffffffffa07a7a05>] handle_tx_kick+0x15/0x20 [vhost_net]
[<ffffffffa07a385a>] vhost_worker+0xfa/0x1a0 [vhost_net]
[<ffffffff8105ef80>] kthread+0xc0/0xd0
[<ffffffff8140399c>] ret_from_fork+0x7c/0xb0
Code: 8b 6d f8 c9 c3 48 8b 07 f6 c4 80 75 0d f0 ff 4b 1c 0f 94 c0 84 c0 74 c9
eb bf 4c 8b 67 30 48 8b 07 f6 c4 80 74 e7 4c 39 e7 74 e2 <41> 8b 54 24 1c 49 8d
4c 24 1c 85 d2 74 d4 8d 72 01 89 d0 f0 0f 
RIP  [<ffffffff810f73a9>] put_compound_page+0x89/0x170
RSP <ffff880036887bc8>
CR2: 000000000000001c
---[ end trace 97d0c820ff95bb21 ]---

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 60518] Heavy network traffic between guest and host triggers kernel oops

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=60518

Bart Van Assche <bvanassche@acm.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Regression|No                          |Yes

--- Comment #1 from Bart Van Assche <bvanassche@acm.org> ---
This does not occur with kernel 3.8.12, so marking this bug as a regression.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 60518] Heavy network traffic between guest and host triggers kernel oops

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=60518

--- Comment #2 from Bart Van Assche <bvanassche@acm.org> ---
Note: so far I haven't been able to trigger this issue with kernel 3.8.12 nor
with kernel 3.10.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 60518] Heavy network traffic between guest and host triggers kernel oops

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=60518

Tim Käck <timkack@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |timkack@gmail.com

--- Comment #3 from Tim Käck <timkack@gmail.com> ---
This does unfortunately still happens in 3.10

Linux ArchSTAR 3.10.3-1-ARCH #1 SMP PREEMPT Fri Jul 26 11:26:59 CEST 2013
x86_64 GNU/Linux

Relevant oops trace:
BUG: unable to handle kernel NULL pointer dereference at           (null)
 IP: [<ffffffff8112eea5>] put_page+0x5/0x40
 PGD 0 
 Oops: 0000 [#1] PREEMPT SMP 
 Modules linked in: vhost_net macvtap macvlan fuse joydev hid_apple hidp
ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf
  snd_timer pci_hotplug parport ptp pps_core libphy lpc_ich snd i5k_amb
soundcore button processor pci_stub vboxpci(O) vboxnetflt(O) vbo..._mod mptbase
 CPU: 3 PID: 4234 Comm: vhost-4233 Tainted: P           O 3.10.3-1-ARCH #1
 Hardware name: Dell Inc. Precision WorkStation T7400  /0RW199, BIOS A04
08/21/2008
 task: ffff880427672a30 ti: ffff8803b9a22000 task.ti: ffff8803b9a22000
 RIP: 0010:[<ffffffff8112eea5>]  [<ffffffff8112eea5>] put_page+0x5/0x40
 RSP: 0018:ffff8803b9a23c70  EFLAGS: 00010202
 RAX: 0000000000000140 RBX: ffff8803996b3ec0 RCX: 0000000000000000
 RDX: ffff8803b9a23fd8 RSI: ffff8803996b3ec0 RDI: 0000000000000000
 RBP: ffff8803b9a23c90 R08: 0000000000000000 R09: 0000000000000000
 R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000012
 R13: ffff8803a7a38700 R14: 000000000000000c R15: ffff8803a7a38700
 FS:  0000000000000000(0000) GS:ffff88043fcc0000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
 CR2: 0000000000000000 CR3: 000000034a32f000 CR4: 00000000000427e0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 Stack:
  ffffffff813b8f38 ffff8803a7a38700 ffff8803a7a38700 ffffffffa005f1b4
  ffff8803b9a23ca8 ffffffff813b8fe4 ffff8803be588800 ffff8803b9a23cd0
  ffffffff813b9046 ffff8803be588800 000000000000b38a 000000000000b396
 Call Trace:
  [<ffffffff813b8f38>] ? skb_release_data+0x88/0x110
  [<ffffffffa005f1b4>] ? tun_get_user+0x724/0x800 [tun]
  [<ffffffff813b8fe4>] skb_release_all+0x24/0x30
  [<ffffffff813b9046>] kfree_skb+0x36/0xe0
  [<ffffffffa005f1b4>] tun_get_user+0x724/0x800 [tun]
  [<ffffffffa005f2e4>] tun_sendmsg+0x54/0x80 [tun]
  [<ffffffffa016aa00>] handle_tx+0x180/0x560 [vhost_net]
  [<ffffffffa016ae15>] handle_tx_kick+0x15/0x20 [vhost_net]
  [<ffffffffa0167437>] vhost_worker+0xf7/0x1a0 [vhost_net]
  [<ffffffffa0167340>] ? vhost_work_flush+0x100/0x100 [vhost_net]
  [<ffffffff8107a610>] kthread+0xc0/0xd0
  [<ffffffff8107a550>] ? kthread_create_on_node+0x120/0x120
  [<ffffffff814ca6ac>] ret_from_fork+0x7c/0xb0
  [<ffffffff8107a550>] ? kthread_create_on_node+0x120/0x120
 Code: d0 f3 90 49 8b 0c 24 f7 c1 00 00 00 01 75 f2 83 80 44 e0 ff ff 01 e9 f1
fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 ...0f 94 c0 84 
 RIP  [<ffffffff8112eea5>] put_page+0x5/0x40
  RSP <ffff8803b9a23c70>
 CR2: 0000000000000000
 ---[ end trace df427221626605d8 ]---

Reproduction steps (almost 100% of the time): 
1) set guest A to be NFS server
2) create deep folder structure with lots of tiny files and share it /share
3) mount on guest B and on host
4) Copy 2 Gb file from Guest B to /share
5) At the same time do "find /share -print > /dev/null' on host

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 60518] Heavy network traffic between guest and host triggers kernel oops

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=60518

--- Comment #4 from Tim Käck <timkack@gmail.com> ---
The problem is gone in 3.10.5.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 60518] Heavy network traffic between guest and host triggers kernel oops

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=60518

Bart Van Assche <bvanassche@acm.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #5 from Bart Van Assche <bvanassche@acm.org> ---
Same results on my setup - 3.10.5 passes my tests.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.