Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover. [1]

Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover. [1]

[David Hill]

Hi guys,

    Starting with kernel 4.14-rc1, my CI failed to completely shutdown 
one of the
VMs and it stuck in "in shutdown" while sending this kernel message:

[ 7496.552971] INFO: task qemu-system-x86:5978 blocked for more than 120 
seconds.
[ 7496.552987]       Tainted: G          I 
4.14.0-0.rc1.git3.1.fc28.x86_64 #1
[ 7496.552996] "echo 0 /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
[ 7496.553006] qemu-system-x86 D12240  5978      1 0x00000004
[ 7496.553024] Call Trace:
[ 7496.553044]  __schedule+0x2dc/0xbb0
[ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
[ 7496.553074]  schedule+0x3d/0x90
[ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
[ 7496.553100]  ? finish_wait+0x90/0x90
[ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
[ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
[ 7496.553166]  SyS_ioctl+0x79/0x90
[ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 7496.553190] RIP: 0033:0x7fa1ea0e1817
[ 7496.553196] RSP: 002b:00007ffe3d854bc8 EFLAGS: 00000246 ORIG_RAX: 
0000000000000010
[ 7496.553209] RAX: ffffffffffffffda RBX: 000000000000001d RCX: 
00007fa1ea0e1817
[ 7496.553215] RDX: 00007ffe3d854bd0 RSI: 000000004008af30 RDI: 
0000000000000021
[ 7496.553222] RBP: 000055e33352b610 R08: 000055e33024a6f0 R09: 
000055e330245d92
[ 7496.553228] R10: 000055e33344e7f0 R11: 0000000000000246 R12: 
000055e33351a000
[ 7496.553235] R13: 0000000000000001 R14: 0000000400000000 R15: 
0000000000000000
[ 7496.553284]
                Showing all locks held in the system:
[ 7496.553313] 1 lock held by khungtaskd/161:
[ 7496.553319]  #0:  (tasklist_lock){.+.+}, at: [<ffffffff8111740d>] 
debug_show_all_locks+0x3d/0x1a0
[ 7496.553373] 1 lock held by in:imklog/1194:
[ 7496.553379]  #0:  (&f->f_pos_lock){+.+.}, at: [<ffffffff8130ecfc>] 
__fdget_pos+0x4c/0x60
[ 7496.553541] 1 lock held by qemu-system-x86/5978:
[ 7496.553547]  #0:  (&dev->mutex#3){+.+.}, at: [<ffffffffc077e498>] 
vhost_net_ioctl+0x358/0x910 [vhost_net]



I'm currently bisecting to figure out which commit breaks this but for 
some reasons, when hitting this commit:

# bad: [46d4b68f891bee5d83a32508bfbd9778be6b1b63] Merge tag 
'wireless-drivers-next-for-davem-2017-08-07' of 
git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next
git bisect bad 46d4b68f891bee5d83a32508bfbd9778be6b1b63

the host will not allow SSHd to establish a new session and when 
starting a KVM guest, the host will hard lock.   I'm still bisecting but 
I marked that commit as bad even though perhaps it would be good.
Hopefully, this commit was a bad one and my bisection will pinpoint which
commit broke the kernel.    If you have an idea of which commit might break
the system, please let me know which one I should test first.

Thank you very much,
David Hill

[1] https://bugzilla.kernel.org/show_bug.cgi?id=197861

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover. [1]

[David Hill]

Hi guys,

     The same behavior is seen with 4.15-rc0 :

$ sudo virsh list
  Id    Name                           State
----------------------------------------------------
  2     undercloud-0-pike              in shutdown

$ uname -a
Linux zappa.orion 4.15.0-0.rc0.git1.1.fc28.x86_64 #1 SMP Mon Nov 13 
19:54:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux


$ ps aufxg | grep qemu

qemu      8595 81.6 12.8 22831200 16961240 ?   D    11:44 214:26 
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name 
guest=undercloud-0-pike,debug-threads=on -S -object 
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-undercloud-0-pike/master-key.aes 
-machine pc-i440fx-2.10,accel=kvm,usb=off,vmport=off,dump-guest-core=off 
-cpu Westmere -m 16384 -realtime mlock=off -smp 
4,sockets=4,cores=1,threads=1 -uuid 0de0918d-f67e-426a-91c1-5e38e86c96b3 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-undercloud-0-pike/monitor.sock,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet 
-no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 
-boot menu=on,strict=on -device 
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device 
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 
-device 
ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 
-device 
ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive 
file=/var/lib/jenkins/VMs/undercloud-0-pike.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 
-drive if=none,id=drive-ide0-0-0,readonly=on -device 
ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=3 
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:79:8e:41,bus=pci.0,addr=0x3,bootindex=1 
-netdev tap,fd=31,id=hostnet1,vhost=on,vhostfd=32 -device 
virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:b8:8b:a5,bus=pci.0,addr=0x8 
-chardev pty,id=charserial0 -device 
isa-serial,chardev=charserial0,id=serial0 -chardev 
spicevmc,id=charchannel0,name=vdagent -device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 
-spice 
port=5900,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-migration=on 
-device 
qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device 
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev 
spicevmc,id=charredir0,name=usbredir -device 
usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev 
spicevmc,id=charredir1,name=usbredir -device 
usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

Thank you very much,

David Hill


On 2017-11-14 10:44 AM, Dave Hill wrote:
> I'm not able to bisect further as the system wont boot properly when I 
> reach that commit.
>
> On Mon, Nov 13, 2017 at 10:54 AM, David Hill <dhill@redhat.com 
> <mailto:dhill@redhat.com>> wrote:
>
>
>     Hi guys,
>
>        Starting with kernel 4.14-rc1, my CI failed to completely
>     shutdown one of the
>     VMs and it stuck in "in shutdown" while sending this kernel message:
>
>     [ 7496.552971] INFO: task qemu-system-x86:5978 blocked for more
>     than 120 seconds.
>     [ 7496.552987]       Tainted: G          I
>     4.14.0-0.rc1.git3.1.fc28.x86_64 #1
>     [ 7496.552996] "echo 0 /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [ 7496.553006] qemu-system-x86 D12240  5978      1 0x00000004
>     [ 7496.553024] Call Trace:
>     [ 7496.553044]  __schedule+0x2dc/0xbb0
>     [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>     [ 7496.553074]  schedule+0x3d/0x90
>     [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>     [ 7496.553100]  ? finish_wait+0x90/0x90
>     [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>     [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>     [ 7496.553166]  SyS_ioctl+0x79/0x90
>     [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>     [ 7496.553190] RIP: 0033:0x7fa1ea0e1817
>     [ 7496.553196] RSP: 002b:00007ffe3d854bc8 EFLAGS: 00000246
>     ORIG_RAX: 0000000000000010
>     [ 7496.553209] RAX: ffffffffffffffda RBX: 000000000000001d RCX:
>     00007fa1ea0e1817
>     [ 7496.553215] RDX: 00007ffe3d854bd0 RSI: 000000004008af30 RDI:
>     0000000000000021
>     [ 7496.553222] RBP: 000055e33352b610 R08: 000055e33024a6f0 R09:
>     000055e330245d92
>     [ 7496.553228] R10: 000055e33344e7f0 R11: 0000000000000246 R12:
>     000055e33351a000
>     [ 7496.553235] R13: 0000000000000001 R14: 0000000400000000 R15:
>     0000000000000000
>     [ 7496.553284]
>                    Showing all locks held in the system:
>     [ 7496.553313] 1 lock held by khungtaskd/161:
>     [ 7496.553319]  #0:  (tasklist_lock){.+.+}, at:
>     [<ffffffff8111740d>] debug_show_all_locks+0x3d/0x1a0
>     [ 7496.553373] 1 lock held by in:imklog/1194:
>     [ 7496.553379]  #0:  (&f->f_pos_lock){+.+.}, at:
>     [<ffffffff8130ecfc>] __fdget_pos+0x4c/0x60
>     [ 7496.553541] 1 lock held by qemu-system-x86/5978:
>     [ 7496.553547]  #0:  (&dev->mutex#3){+.+.}, at:
>     [<ffffffffc077e498>] vhost_net_ioctl+0x358/0x910 [vhost_net]
>
>
>
>     I'm currently bisecting to figure out which commit breaks this but
>     for some reasons, when hitting this commit:
>
>     # bad: [46d4b68f891bee5d83a32508bfbd9778be6b1b63] Merge tag
>     'wireless-drivers-next-for-davem-2017-08-07' of
>     git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next
>     <http://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next>
>     git bisect bad 46d4b68f891bee5d83a32508bfbd9778be6b1b63
>
>     the host will not allow SSHd to establish a new session and when
>     starting a KVM guest, the host will hard lock.   I'm still
>     bisecting but I marked that commit as bad even though perhaps it
>     would be good.
>     Hopefully, this commit was a bad one and my bisection will
>     pinpoint which
>     commit broke the kernel.    If you have an idea of which commit
>     might break
>     the system, please let me know which one I should test first.
>
>     Thank you very much,
>     David Hill
>
>     [1] https://bugzilla.kernel.org/show_bug.cgi?id=197861
>     <https://bugzilla.kernel.org/show_bug.cgi?id=197861>
>
>
>

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

Anyone had time to take a little look at the kernel trace?


On 2017-11-15 04:08 PM, David Hill wrote:
> Hi guys,
>
>     The same behavior is seen with 4.15-rc0 :
>
> $ sudo virsh list
>  Id    Name                           State
> ----------------------------------------------------
>  2     undercloud-0-pike              in shutdown
>
> $ uname -a
> Linux zappa.orion 4.15.0-0.rc0.git1.1.fc28.x86_64 #1 SMP Mon Nov 13 
> 19:54:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>
>
> $ ps aufxg | grep qemu
>
> qemu      8595 81.6 12.8 22831200 16961240 ?   D    11:44 214:26 
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name 
> guest=undercloud-0-pike,debug-threads=on -S -object 
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-undercloud-0-pike/master-key.aes 
> -machine 
> pc-i440fx-2.10,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu 
> Westmere -m 16384 -realtime mlock=off -smp 
> 4,sockets=4,cores=1,threads=1 -uuid 
> 0de0918d-f67e-426a-91c1-5e38e86c96b3 -no-user-config -nodefaults 
> -chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-undercloud-0-pike/monitor.sock,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc 
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet 
> -no-shutdown -global PIIX4_PM.disable_s3=1 -global 
> PIIX4_PM.disable_s4=1 -boot menu=on,strict=on -device 
> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device 
> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 
> -device 
> ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 
> -device 
> ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 
> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive 
> file=/var/lib/jenkins/VMs/undercloud-0-pike.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 
> -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 
> -drive if=none,id=drive-ide0-0-0,readonly=on -device 
> ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=3 
> -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=30 -device 
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:79:8e:41,bus=pci.0,addr=0x3,bootindex=1 
> -netdev tap,fd=31,id=hostnet1,vhost=on,vhostfd=32 -device 
> virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:b8:8b:a5,bus=pci.0,addr=0x8 
> -chardev pty,id=charserial0 -device 
> isa-serial,chardev=charserial0,id=serial0 -chardev 
> spicevmc,id=charchannel0,name=vdagent -device 
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 
> -spice 
> port=5900,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-migration=on 
> -device 
> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 
> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device 
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev 
> spicevmc,id=charredir0,name=usbredir -device 
> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev 
> spicevmc,id=charredir1,name=usbredir -device 
> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device 
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
>
> Thank you very much,
>
> David Hill
>
>
> On 2017-11-14 10:44 AM, Dave Hill wrote:
>> I'm not able to bisect further as the system wont boot properly when 
>> I reach that commit.
>>
>> On Mon, Nov 13, 2017 at 10:54 AM, David Hill <dhill@redhat.com 
>> <mailto:dhill@redhat.com>> wrote:
>>
>>
>>     Hi guys,
>>
>>        Starting with kernel 4.14-rc1, my CI failed to completely
>>     shutdown one of the
>>     VMs and it stuck in "in shutdown" while sending this kernel message:
>>
>>     [ 7496.552971] INFO: task qemu-system-x86:5978 blocked for more
>>     than 120 seconds.
>>     [ 7496.552987]       Tainted: G          I
>>     4.14.0-0.rc1.git3.1.fc28.x86_64 #1
>>     [ 7496.552996] "echo 0 /proc/sys/kernel/hung_task_timeout_secs"
>>     disables this message.
>>     [ 7496.553006] qemu-system-x86 D12240  5978      1 0x00000004
>>     [ 7496.553024] Call Trace:
>>     [ 7496.553044]  __schedule+0x2dc/0xbb0
>>     [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>>     [ 7496.553074]  schedule+0x3d/0x90
>>     [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>>     [ 7496.553100]  ? finish_wait+0x90/0x90
>>     [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>>     [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>>     [ 7496.553166]  SyS_ioctl+0x79/0x90
>>     [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>>     [ 7496.553190] RIP: 0033:0x7fa1ea0e1817
>>     [ 7496.553196] RSP: 002b:00007ffe3d854bc8 EFLAGS: 00000246
>>     ORIG_RAX: 0000000000000010
>>     [ 7496.553209] RAX: ffffffffffffffda RBX: 000000000000001d RCX:
>>     00007fa1ea0e1817
>>     [ 7496.553215] RDX: 00007ffe3d854bd0 RSI: 000000004008af30 RDI:
>>     0000000000000021
>>     [ 7496.553222] RBP: 000055e33352b610 R08: 000055e33024a6f0 R09:
>>     000055e330245d92
>>     [ 7496.553228] R10: 000055e33344e7f0 R11: 0000000000000246 R12:
>>     000055e33351a000
>>     [ 7496.553235] R13: 0000000000000001 R14: 0000000400000000 R15:
>>     0000000000000000
>>     [ 7496.553284]
>>                    Showing all locks held in the system:
>>     [ 7496.553313] 1 lock held by khungtaskd/161:
>>     [ 7496.553319]  #0:  (tasklist_lock){.+.+}, at:
>>     [<ffffffff8111740d>] debug_show_all_locks+0x3d/0x1a0
>>     [ 7496.553373] 1 lock held by in:imklog/1194:
>>     [ 7496.553379]  #0:  (&f->f_pos_lock){+.+.}, at:
>>     [<ffffffff8130ecfc>] __fdget_pos+0x4c/0x60
>>     [ 7496.553541] 1 lock held by qemu-system-x86/5978:
>>     [ 7496.553547]  #0:  (&dev->mutex#3){+.+.}, at:
>>     [<ffffffffc077e498>] vhost_net_ioctl+0x358/0x910 [vhost_net]
>>
>>
>>
>>     I'm currently bisecting to figure out which commit breaks this but
>>     for some reasons, when hitting this commit:
>>
>>     # bad: [46d4b68f891bee5d83a32508bfbd9778be6b1b63] Merge tag
>>     'wireless-drivers-next-for-davem-2017-08-07' of
>> git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next
>> <http://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next>
>>     git bisect bad 46d4b68f891bee5d83a32508bfbd9778be6b1b63
>>
>>     the host will not allow SSHd to establish a new session and when
>>     starting a KVM guest, the host will hard lock.   I'm still
>>     bisecting but I marked that commit as bad even though perhaps it
>>     would be good.
>>     Hopefully, this commit was a bad one and my bisection will
>>     pinpoint which
>>     commit broke the kernel.    If you have an idea of which commit
>>     might break
>>     the system, please let me know which one I should test first.
>>
>>     Thank you very much,
>>     David Hill
>>
>>     [1] https://bugzilla.kernel.org/show_bug.cgi?id=197861
>>     <https://bugzilla.kernel.org/show_bug.cgi?id=197861>
>>
>>
>>
>

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Paolo Bonzini]

Jason, any ideas?

Thanks,

Paolo

On 22/11/2017 19:22, David Hill wrote:
> ore than 120 seconds.
>     [ 7496.552987]       Tainted: G          I
>     4.14.0-0.rc1.git3.1.fc28.x86_64 #1
>     [ 7496.552996] "echo 0 /proc/sys/kernel/hung_task_timeout_secs"
>     disables this message.
>     [ 7496.553006] qemu-system-x86 D12240  5978      1 0x00000004
>     [ 7496.553024] Call Trace:
>     [ 7496.553044]  __schedule+0x2dc/0xbb0
>     [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>     [ 7496.553074]  schedule+0x3d/0x90
>     [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>     [ 7496.553100]  ? finish_wait+0x90/0x90
>     [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>     [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>     [ 7496.553166]  SyS_ioctl+0x79/0x90
>     [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>     [ 7496.553190] RIP: 0033:0x7fa1ea0e1817
>     [ 7496.553196] RSP: 002b:00007ffe3d854bc8 EFLAGS: 00000246
>     ORIG_RAX: 0000000000000010
>     [ 7496.553209] RAX: ffffffffffffffda RBX: 000000000000001d RCX:
>     00007fa1ea0e1817
>     [ 7496.553215] RDX: 00007ffe3d854bd0 RSI: 000000004008af30 RDI:
>     0000000000000021
>     [ 7496.553222] RBP: 000055e33352b610 R08: 000055e33024a6f0 R09:
>     000055e330245d92
>     [ 7496.553228] R10: 000055e33344e7f0 R11: 0000000000000246 R12:
>     000055e33351a000
>     [ 7496.553235] R13: 0000000000000001 R14: 0000000400000000 R15:
>     0000000000000000
>     [ 7496.553284]
>                    Showing all locks held in the system:
>     [ 7496.553313] 1 lock held by khungtaskd/161:
>     [ 7496.553319]  #0:  (tasklist_lock){.+.+}, at:
>     [<ffffffff8111740d>] debug_show_all_locks+0x3d/0x1a0
>     [ 7496.553373] 1 lock held by in:imklog/1194:
>     [ 7496.553379]  #0:  (&f->f_pos_lock){+.+.}, at:
>     [<ffffffff8130ecfc>] __fdget_pos+0x4c/0x60
>     [ 7496.553541] 1 lock held by qemu-system-x86/5978:
>     [ 7496.553547]  #0:  (&dev->mutex#3){+.+.}, at:
>     [<ffffffffc077e498>] vhost_net_ioctl+0x358/0x910 [vhost_net]

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年11月24日 07:48, Paolo Bonzini wrote:
> Jason, any ideas?
>
> Thanks,
>
> Paolo
>
> On 22/11/2017 19:22, David Hill wrote:
>> ore than 120 seconds.
>>      [ 7496.552987]       Tainted: G          I
>>      4.14.0-0.rc1.git3.1.fc28.x86_64 #1
>>      [ 7496.552996] "echo 0 /proc/sys/kernel/hung_task_timeout_secs"
>>      disables this message.
>>      [ 7496.553006] qemu-system-x86 D12240  5978      1 0x00000004
>>      [ 7496.553024] Call Trace:
>>      [ 7496.553044]  __schedule+0x2dc/0xbb0
>>      [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>>      [ 7496.553074]  schedule+0x3d/0x90
>>      [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>>      [ 7496.553100]  ? finish_wait+0x90/0x90
>>      [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>>      [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>>      [ 7496.553166]  SyS_ioctl+0x79/0x90
>>      [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>>      [ 7496.553190] RIP: 0033:0x7fa1ea0e1817
>>      [ 7496.553196] RSP: 002b:00007ffe3d854bc8 EFLAGS: 00000246
>>      ORIG_RAX: 0000000000000010
>>      [ 7496.553209] RAX: ffffffffffffffda RBX: 000000000000001d RCX:
>>      00007fa1ea0e1817
>>      [ 7496.553215] RDX: 00007ffe3d854bd0 RSI: 000000004008af30 RDI:
>>      0000000000000021
>>      [ 7496.553222] RBP: 000055e33352b610 R08: 000055e33024a6f0 R09:
>>      000055e330245d92
>>      [ 7496.553228] R10: 000055e33344e7f0 R11: 0000000000000246 R12:
>>      000055e33351a000
>>      [ 7496.553235] R13: 0000000000000001 R14: 0000000400000000 R15:
>>      0000000000000000
>>      [ 7496.553284]
>>                     Showing all locks held in the system:
>>      [ 7496.553313] 1 lock held by khungtaskd/161:
>>      [ 7496.553319]  #0:  (tasklist_lock){.+.+}, at:
>>      [<ffffffff8111740d>] debug_show_all_locks+0x3d/0x1a0
>>      [ 7496.553373] 1 lock held by in:imklog/1194:
>>      [ 7496.553379]  #0:  (&f->f_pos_lock){+.+.}, at:
>>      [<ffffffff8130ecfc>] __fdget_pos+0x4c/0x60
>>      [ 7496.553541] 1 lock held by qemu-system-x86/5978:
>>      [ 7496.553547]  #0:  (&dev->mutex#3){+.+.}, at:
>>      [<ffffffffc077e498>] vhost_net_ioctl+0x358/0x910 [vhost_net]

Hi:

The backtrace shows zero copied skb was not sent for a long while for 
some reason. This could be either a bug in vhost_net or somewhere in the 
host driver, qdiscs or others.

What's your network setups in host (e.g the qdiscs or network driver)? 
Can you still hit the issue if you switch to use another type of 
ethernet driver/cards? Can this still be reproducible in net.git 
(https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/).

Will try to reproduce this locally.

Thanks

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

It's really hard to reproduce actually because in order to hit this 
issue, I must do the following:


1) Create an undercloud VM (tripleo openstack)

2) Update it, install an undercloud

3) Create 7 other VMs

4) Deploy an overcloud on these 7 VMs

5) Stop those VMs

6) Try stopping the undercloud VM and it get stucks "in shutdown"


It breaks here ... I tried reproducing this by starting/stopping VMs and 
I wasn't able to hit it ...   It takes a good 2-3 hours each time I test 
this.

I get this issue only with 4.14 and later and I wasn't able to reproduce 
this with 4.13 and older.


On 2017-11-23 10:11 PM, Jason Wang wrote:
>
>
> On 2017年11月24日 07:48, Paolo Bonzini wrote:
>> Jason, any ideas?
>>
>> Thanks,
>>
>> Paolo
>>
>> On 22/11/2017 19:22, David Hill wrote:
>>> ore than 120 seconds.
>>>      [ 7496.552987]       Tainted: G          I
>>>      4.14.0-0.rc1.git3.1.fc28.x86_64 #1
>>>      [ 7496.552996] "echo 0 /proc/sys/kernel/hung_task_timeout_secs"
>>>      disables this message.
>>>      [ 7496.553006] qemu-system-x86 D12240  5978      1 0x00000004
>>>      [ 7496.553024] Call Trace:
>>>      [ 7496.553044]  __schedule+0x2dc/0xbb0
>>>      [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>>>      [ 7496.553074]  schedule+0x3d/0x90
>>>      [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>>>      [ 7496.553100]  ? finish_wait+0x90/0x90
>>>      [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>>>      [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>>>      [ 7496.553166]  SyS_ioctl+0x79/0x90
>>>      [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>>>      [ 7496.553190] RIP: 0033:0x7fa1ea0e1817
>>>      [ 7496.553196] RSP: 002b:00007ffe3d854bc8 EFLAGS: 00000246
>>>      ORIG_RAX: 0000000000000010
>>>      [ 7496.553209] RAX: ffffffffffffffda RBX: 000000000000001d RCX:
>>>      00007fa1ea0e1817
>>>      [ 7496.553215] RDX: 00007ffe3d854bd0 RSI: 000000004008af30 RDI:
>>>      0000000000000021
>>>      [ 7496.553222] RBP: 000055e33352b610 R08: 000055e33024a6f0 R09:
>>>      000055e330245d92
>>>      [ 7496.553228] R10: 000055e33344e7f0 R11: 0000000000000246 R12:
>>>      000055e33351a000
>>>      [ 7496.553235] R13: 0000000000000001 R14: 0000000400000000 R15:
>>>      0000000000000000
>>>      [ 7496.553284]
>>>                     Showing all locks held in the system:
>>>      [ 7496.553313] 1 lock held by khungtaskd/161:
>>>      [ 7496.553319]  #0:  (tasklist_lock){.+.+}, at:
>>>      [<ffffffff8111740d>] debug_show_all_locks+0x3d/0x1a0
>>>      [ 7496.553373] 1 lock held by in:imklog/1194:
>>>      [ 7496.553379]  #0:  (&f->f_pos_lock){+.+.}, at:
>>>      [<ffffffff8130ecfc>] __fdget_pos+0x4c/0x60
>>>      [ 7496.553541] 1 lock held by qemu-system-x86/5978:
>>>      [ 7496.553547]  #0:  (&dev->mutex#3){+.+.}, at:
>>>      [<ffffffffc077e498>] vhost_net_ioctl+0x358/0x910 [vhost_net]
>
> Hi:
>
> The backtrace shows zero copied skb was not sent for a long while for 
> some reason. This could be either a bug in vhost_net or somewhere in 
> the host driver, qdiscs or others.
>
> What's your network setups in host (e.g the qdiscs or network driver)? 
> Can you still hit the issue if you switch to use another type of 
> ethernet driver/cards? Can this still be reproducible in net.git 
> (https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/).
>
> Will try to reproduce this locally.
>
> Thanks

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

The VMs all have 2 vNICs ... and this is the hypervisor:

[root@zappa ~]# brctl show
bridge name    bridge id        STP enabled    interfaces
virbr0        8000.525400914858    yes        virbr0-nic
                             vnet0
                             vnet1


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
group default qlen 1000
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
group default qlen 1000
     link/ether 84:2b:2b:13:f2:91 brd ff:ff:ff:ff:ff:ff
     inet redacted/24 brd 173.178.138.255 scope global dynamic eno1
        valid_lft 48749sec preferred_lft 48749sec
     inet6 fe80::862b:2bff:fe13:f291/64 scope link
        valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
group default qlen 1000
     link/ether 84:2b:2b:13:f2:92 brd ff:ff:ff:ff:ff:ff
     inet 192.168.1.3/24 brd 192.168.1.255 scope global eno2
        valid_lft forever preferred_lft forever
     inet6 fe80::862b:2bff:fe13:f292/64 scope link
        valid_lft forever preferred_lft forever
4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
state UP group default qlen 1000
     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

        valid_lft forever preferred_lft forever
     inet 192.168.122.10/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.11/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.12/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.15/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.16/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.17/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.18/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.31/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.32/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.33/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.34/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.35/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.36/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.37/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.45/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.46/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.47/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.48/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.49/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.50/32 scope global virbr0
        valid_lft forever preferred_lft forever
     inet 192.168.122.51/32 scope global virbr0
        valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master 
virbr0 state DOWN group default qlen 1000
     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
125: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 qdisc 
fq_codel state UNKNOWN group default qlen 100
     link/none
     inet 10.10.122.28/21 brd 10.10.127.255 scope global tun0
        valid_lft forever preferred_lft forever
     inet6 fe80::1f9b:bfd4:e9c9:2059/64 scope link stable-privacy
        valid_lft forever preferred_lft forever
402: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
master virbr0 state UNKNOWN group default qlen 1000
     link/ether fe:54:00:09:27:39 brd ff:ff:ff:ff:ff:ff
     inet6 fe80::fc54:ff:fe09:2739/64 scope link
        valid_lft forever preferred_lft forever
403: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
master virbr0 state UNKNOWN group default qlen 1000
     link/ether fe:54:00:ea:6b:18 brd ff:ff:ff:ff:ff:ff
     inet6 fe80::fc54:ff:feea:6b18/64 scope link
        valid_lft forever preferred_lft forever


On 2017-11-23 10:11 PM, Jason Wang wrote:
>
>
> On 2017年11月24日 07:48, Paolo Bonzini wrote:
>> Jason, any ideas?
>>
>> Thanks,
>>
>> Paolo
>>
>> On 22/11/2017 19:22, David Hill wrote:
>>> ore than 120 seconds.
>>>      [ 7496.552987]       Tainted: G          I
>>>      4.14.0-0.rc1.git3.1.fc28.x86_64 #1
>>>      [ 7496.552996] "echo 0 /proc/sys/kernel/hung_task_timeout_secs"
>>>      disables this message.
>>>      [ 7496.553006] qemu-system-x86 D12240  5978      1 0x00000004
>>>      [ 7496.553024] Call Trace:
>>>      [ 7496.553044]  __schedule+0x2dc/0xbb0
>>>      [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>>>      [ 7496.553074]  schedule+0x3d/0x90
>>>      [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>>>      [ 7496.553100]  ? finish_wait+0x90/0x90
>>>      [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>>>      [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>>>      [ 7496.553166]  SyS_ioctl+0x79/0x90
>>>      [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>>>      [ 7496.553190] RIP: 0033:0x7fa1ea0e1817
>>>      [ 7496.553196] RSP: 002b:00007ffe3d854bc8 EFLAGS: 00000246
>>>      ORIG_RAX: 0000000000000010
>>>      [ 7496.553209] RAX: ffffffffffffffda RBX: 000000000000001d RCX:
>>>      00007fa1ea0e1817
>>>      [ 7496.553215] RDX: 00007ffe3d854bd0 RSI: 000000004008af30 RDI:
>>>      0000000000000021
>>>      [ 7496.553222] RBP: 000055e33352b610 R08: 000055e33024a6f0 R09:
>>>      000055e330245d92
>>>      [ 7496.553228] R10: 000055e33344e7f0 R11: 0000000000000246 R12:
>>>      000055e33351a000
>>>      [ 7496.553235] R13: 0000000000000001 R14: 0000000400000000 R15:
>>>      0000000000000000
>>>      [ 7496.553284]
>>>                     Showing all locks held in the system:
>>>      [ 7496.553313] 1 lock held by khungtaskd/161:
>>>      [ 7496.553319]  #0:  (tasklist_lock){.+.+}, at:
>>>      [<ffffffff8111740d>] debug_show_all_locks+0x3d/0x1a0
>>>      [ 7496.553373] 1 lock held by in:imklog/1194:
>>>      [ 7496.553379]  #0:  (&f->f_pos_lock){+.+.}, at:
>>>      [<ffffffff8130ecfc>] __fdget_pos+0x4c/0x60
>>>      [ 7496.553541] 1 lock held by qemu-system-x86/5978:
>>>      [ 7496.553547]  #0:  (&dev->mutex#3){+.+.}, at:
>>>      [<ffffffffc077e498>] vhost_net_ioctl+0x358/0x910 [vhost_net]
>
> Hi:
>
> The backtrace shows zero copied skb was not sent for a long while for 
> some reason. This could be either a bug in vhost_net or somewhere in 
> the host driver, qdiscs or others.
>
> What's your network setups in host (e.g the qdiscs or network driver)? 
> Can you still hit the issue if you switch to use another type of 
> ethernet driver/cards? Can this still be reproducible in net.git 
> (https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/).
>
> Will try to reproduce this locally.
>
> Thanks

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年11月25日 00:22, David Hill wrote:
> The VMs all have 2 vNICs ... and this is the hypervisor:
>
> [root@zappa ~]# brctl show
> bridge name    bridge id        STP enabled    interfaces
> virbr0        8000.525400914858    yes        virbr0-nic
>                             vnet0
>                             vnet1
>
>
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
> group default qlen 1000
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>        valid_lft forever preferred_lft forever
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
> 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
> group default qlen 1000
>     link/ether 84:2b:2b:13:f2:91 brd ff:ff:ff:ff:ff:ff
>     inet redacted/24 brd 173.178.138.255 scope global dynamic eno1
>        valid_lft 48749sec preferred_lft 48749sec
>     inet6 fe80::862b:2bff:fe13:f291/64 scope link
>        valid_lft forever preferred_lft forever
> 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
> group default qlen 1000
>     link/ether 84:2b:2b:13:f2:92 brd ff:ff:ff:ff:ff:ff
>     inet 192.168.1.3/24 brd 192.168.1.255 scope global eno2
>        valid_lft forever preferred_lft forever
>     inet6 fe80::862b:2bff:fe13:f292/64 scope link
>        valid_lft forever preferred_lft forever
> 4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
> state UP group default qlen 1000
>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
>
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.10/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.11/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.12/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.15/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.16/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.17/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.18/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.31/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.32/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.33/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.34/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.35/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.36/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.37/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.45/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.46/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.47/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.48/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.49/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.50/32 scope global virbr0
>        valid_lft forever preferred_lft forever
>     inet 192.168.122.51/32 scope global virbr0
>        valid_lft forever preferred_lft forever
> 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master 
> virbr0 state DOWN group default qlen 1000
>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
> 125: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 qdisc 
> fq_codel state UNKNOWN group default qlen 100
>     link/none
>     inet 10.10.122.28/21 brd 10.10.127.255 scope global tun0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::1f9b:bfd4:e9c9:2059/64 scope link stable-privacy
>        valid_lft forever preferred_lft forever
> 402: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
> master virbr0 state UNKNOWN group default qlen 1000
>     link/ether fe:54:00:09:27:39 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::fc54:ff:fe09:2739/64 scope link
>        valid_lft forever preferred_lft forever
> 403: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
> master virbr0 state UNKNOWN group default qlen 1000
>     link/ether fe:54:00:ea:6b:18 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::fc54:ff:feea:6b18/64 scope link
>        valid_lft forever preferred_lft forever
>

I could not reproduce this locally by simply running netperf through a 
mlx4 card. Some more questions:

- What kind of workloads did you run in guest?
- Did you meet this issue in a specific type of network card (I guess 
broadcom is used in this case)?
- Virbr0 looks like a bridge created by libvirt that did NAT and other 
stuffs, can you still hit this issue if you don't use virbr0?

And what's more important, zerocopy is known to have issues, for 
production environment, need to disable it through vhost_net module 
parameters.

Thanks

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-11-26 10:44 PM, Jason Wang wrote:
>
>
> On 2017年11月25日 00:22, David Hill wrote:
>> The VMs all have 2 vNICs ... and this is the hypervisor:
>>
>> [root@zappa ~]# brctl show
>> bridge name    bridge id        STP enabled    interfaces
>> virbr0        8000.525400914858    yes        virbr0-nic
>>                             vnet0
>>                             vnet1
>>
>>
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
>> group default qlen 1000
>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>     inet 127.0.0.1/8 scope host lo
>>        valid_lft forever preferred_lft forever
>>     inet6 ::1/128 scope host
>>        valid_lft forever preferred_lft forever
>> 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
>> group default qlen 1000
>>     link/ether 84:2b:2b:13:f2:91 brd ff:ff:ff:ff:ff:ff
>>     inet redacted/24 brd 173.178.138.255 scope global dynamic eno1
>>        valid_lft 48749sec preferred_lft 48749sec
>>     inet6 fe80::862b:2bff:fe13:f291/64 scope link
>>        valid_lft forever preferred_lft forever
>> 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
>> group default qlen 1000
>>     link/ether 84:2b:2b:13:f2:92 brd ff:ff:ff:ff:ff:ff
>>     inet 192.168.1.3/24 brd 192.168.1.255 scope global eno2
>>        valid_lft forever preferred_lft forever
>>     inet6 fe80::862b:2bff:fe13:f292/64 scope link
>>        valid_lft forever preferred_lft forever
>> 4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
>> state UP group default qlen 1000
>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>>     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
>>
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.10/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.11/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.12/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.15/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.16/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.17/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.18/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.31/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.32/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.33/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.34/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.35/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.36/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.37/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.45/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.46/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.47/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.48/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.49/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.50/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>>     inet 192.168.122.51/32 scope global virbr0
>>        valid_lft forever preferred_lft forever
>> 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master 
>> virbr0 state DOWN group default qlen 1000
>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>> 125: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 qdisc 
>> fq_codel state UNKNOWN group default qlen 100
>>     link/none
>>     inet 10.10.122.28/21 brd 10.10.127.255 scope global tun0
>>        valid_lft forever preferred_lft forever
>>     inet6 fe80::1f9b:bfd4:e9c9:2059/64 scope link stable-privacy
>>        valid_lft forever preferred_lft forever
>> 402: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
>> master virbr0 state UNKNOWN group default qlen 1000
>>     link/ether fe:54:00:09:27:39 brd ff:ff:ff:ff:ff:ff
>>     inet6 fe80::fc54:ff:fe09:2739/64 scope link
>>        valid_lft forever preferred_lft forever
>> 403: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel 
>> master virbr0 state UNKNOWN group default qlen 1000
>>     link/ether fe:54:00:ea:6b:18 brd ff:ff:ff:ff:ff:ff
>>     inet6 fe80::fc54:ff:feea:6b18/64 scope link
>>        valid_lft forever preferred_lft forever
>>
>
> I could not reproduce this locally by simply running netperf through a 
> mlx4 card. Some more questions:
>
> - What kind of workloads did you run in guest?
> - Did you meet this issue in a specific type of network card (I guess 
> broadcom is used in this case)?
> - Virbr0 looks like a bridge created by libvirt that did NAT and other 
> stuffs, can you still hit this issue if you don't use virbr0?
>
> And what's more important, zerocopy is known to have issues, for 
> production environment, need to disable it through vhost_net module 
> parameters.
>
> Thanks

I'm deploying an overcloud through a undercloud virtual machine... The 
VM has 4vCPUs and 16GB of RAM as well as to virtio nics so I'm using 
only virtual hardware here.
I spawn 7 VMs on the hypervisor and deploy an overcloud using tripleo on 
them ... everything's virtual and if I remove the bridge, then I'll have 
to configure each VMs differently.
The load is quite high on the VM that won't shutdown but when I shut it 
down, it's doing nothing ...   This is a hard bug to troubleshoot and I 
can't bisect the kernel because at some
point the system simply won't boot properly.

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-11-27 02:38 PM, David Hill wrote:
>
>
> On 2017-11-26 10:44 PM, Jason Wang wrote:
>>
>>
>> On 2017年11月25日 00:22, David Hill wrote:
>>> The VMs all have 2 vNICs ... and this is the hypervisor:
>>>
>>> [root@zappa ~]# brctl show
>>> bridge name    bridge id        STP enabled    interfaces
>>> virbr0        8000.525400914858    yes        virbr0-nic
>>>                             vnet0
>>>                             vnet1
>>>
>>>
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
>>> group default qlen 1000
>>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>>     inet 127.0.0.1/8 scope host lo
>>>        valid_lft forever preferred_lft forever
>>>     inet6 ::1/128 scope host
>>>        valid_lft forever preferred_lft forever
>>> 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
>>> UP group default qlen 1000
>>>     link/ether 84:2b:2b:13:f2:91 brd ff:ff:ff:ff:ff:ff
>>>     inet redacted/24 brd 173.178.138.255 scope global dynamic eno1
>>>        valid_lft 48749sec preferred_lft 48749sec
>>>     inet6 fe80::862b:2bff:fe13:f291/64 scope link
>>>        valid_lft forever preferred_lft forever
>>> 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
>>> UP group default qlen 1000
>>>     link/ether 84:2b:2b:13:f2:92 brd ff:ff:ff:ff:ff:ff
>>>     inet 192.168.1.3/24 brd 192.168.1.255 scope global eno2
>>>        valid_lft forever preferred_lft forever
>>>     inet6 fe80::862b:2bff:fe13:f292/64 scope link
>>>        valid_lft forever preferred_lft forever
>>> 4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
>>> state UP group default qlen 1000
>>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>>>     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
>>>
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.10/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.11/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.12/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.15/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.16/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.17/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.18/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.31/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.32/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.33/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.34/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.35/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.36/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.37/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.45/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.46/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.47/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.48/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.49/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.50/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>>     inet 192.168.122.51/32 scope global virbr0
>>>        valid_lft forever preferred_lft forever
>>> 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master 
>>> virbr0 state DOWN group default qlen 1000
>>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>>> 125: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 qdisc 
>>> fq_codel state UNKNOWN group default qlen 100
>>>     link/none
>>>     inet 10.10.122.28/21 brd 10.10.127.255 scope global tun0
>>>        valid_lft forever preferred_lft forever
>>>     inet6 fe80::1f9b:bfd4:e9c9:2059/64 scope link stable-privacy
>>>        valid_lft forever preferred_lft forever
>>> 402: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
>>> fq_codel master virbr0 state UNKNOWN group default qlen 1000
>>>     link/ether fe:54:00:09:27:39 brd ff:ff:ff:ff:ff:ff
>>>     inet6 fe80::fc54:ff:fe09:2739/64 scope link
>>>        valid_lft forever preferred_lft forever
>>> 403: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
>>> fq_codel master virbr0 state UNKNOWN group default qlen 1000
>>>     link/ether fe:54:00:ea:6b:18 brd ff:ff:ff:ff:ff:ff
>>>     inet6 fe80::fc54:ff:feea:6b18/64 scope link
>>>        valid_lft forever preferred_lft forever
>>>
>>
>> I could not reproduce this locally by simply running netperf through 
>> a mlx4 card. Some more questions:
>>
>> - What kind of workloads did you run in guest?
>> - Did you meet this issue in a specific type of network card (I guess 
>> broadcom is used in this case)?
>> - Virbr0 looks like a bridge created by libvirt that did NAT and 
>> other stuffs, can you still hit this issue if you don't use virbr0?
>>
>> And what's more important, zerocopy is known to have issues, for 
>> production environment, need to disable it through vhost_net module 
>> parameters.
>>
>> Thanks
>
> I'm deploying an overcloud through a undercloud virtual machine... The 
> VM has 4vCPUs and 16GB of RAM as well as to virtio nics so I'm using 
> only virtual hardware here.
> I spawn 7 VMs on the hypervisor and deploy an overcloud using tripleo 
> on them ... everything's virtual and if I remove the bridge, then I'll 
> have to configure each VMs differently.
> The load is quite high on the VM that won't shutdown but when I shut 
> it down, it's doing nothing ...   This is a hard bug to troubleshoot 
> and I can't bisect the kernel because at some
> point the system simply won't boot properly.

I've disabled zerocopy with the following:

[root@zappa modprobe.d]# cat vhost-net.conf
options vhost_net  experimental_zcopytx=0


And I haven't reproduce this issue so far.   The problem I have right 
now is that experimental_zcopytx has been enabled by default with this 
commit:

commit f9611c43ab0ddaf547b395c90fb842f55959334c
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Thu Dec 6 14:56:00 2012 +0200

     vhost-net: enable zerocopy tx by default

     Zero copy TX has been around for a while now.
     We seem to be down to eliminating theoretical bugs
     and performance tuning at this point:
     it's probably time to enable it by default so that
     most users get the benefit.

     Keep the flag around meanwhile so users can experiment
     with disabling this if they experience regressions.
     I expect that we will remove it in the future.

     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

I'll try some more pass in producing this issue and I'll keep you posted.

Thank you very much,

David Hill

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年11月29日 02:00, David Hill wrote:
>
>
> On 2017-11-27 02:38 PM, David Hill wrote:
>>
>>
>> On 2017-11-26 10:44 PM, Jason Wang wrote:
>>>
>>>
>>> On 2017年11月25日 00:22, David Hill wrote:
>>>> The VMs all have 2 vNICs ... and this is the hypervisor:
>>>>
>>>> [root@zappa ~]# brctl show
>>>> bridge name    bridge id        STP enabled    interfaces
>>>> virbr0        8000.525400914858    yes        virbr0-nic
>>>>                             vnet0
>>>>                             vnet1
>>>>
>>>>
>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
>>>> group default qlen 1000
>>>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>>>     inet 127.0.0.1/8 scope host lo
>>>>        valid_lft forever preferred_lft forever
>>>>     inet6 ::1/128 scope host
>>>>        valid_lft forever preferred_lft forever
>>>> 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
>>>> UP group default qlen 1000
>>>>     link/ether 84:2b:2b:13:f2:91 brd ff:ff:ff:ff:ff:ff
>>>>     inet redacted/24 brd 173.178.138.255 scope global dynamic eno1
>>>>        valid_lft 48749sec preferred_lft 48749sec
>>>>     inet6 fe80::862b:2bff:fe13:f291/64 scope link
>>>>        valid_lft forever preferred_lft forever
>>>> 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
>>>> UP group default qlen 1000
>>>>     link/ether 84:2b:2b:13:f2:92 brd ff:ff:ff:ff:ff:ff
>>>>     inet 192.168.1.3/24 brd 192.168.1.255 scope global eno2
>>>>        valid_lft forever preferred_lft forever
>>>>     inet6 fe80::862b:2bff:fe13:f292/64 scope link
>>>>        valid_lft forever preferred_lft forever
>>>> 4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
>>>> state UP group default qlen 1000
>>>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>>>>     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
>>>>
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.10/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.11/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.12/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.15/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.16/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.17/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.18/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.31/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.32/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.33/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.34/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.35/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.36/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.37/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.45/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.46/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.47/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.48/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.49/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.50/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet 192.168.122.51/32 scope global virbr0
>>>>        valid_lft forever preferred_lft forever
>>>> 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master 
>>>> virbr0 state DOWN group default qlen 1000
>>>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>>>> 125: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 qdisc 
>>>> fq_codel state UNKNOWN group default qlen 100
>>>>     link/none
>>>>     inet 10.10.122.28/21 brd 10.10.127.255 scope global tun0
>>>>        valid_lft forever preferred_lft forever
>>>>     inet6 fe80::1f9b:bfd4:e9c9:2059/64 scope link stable-privacy
>>>>        valid_lft forever preferred_lft forever
>>>> 402: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
>>>> fq_codel master virbr0 state UNKNOWN group default qlen 1000
>>>>     link/ether fe:54:00:09:27:39 brd ff:ff:ff:ff:ff:ff
>>>>     inet6 fe80::fc54:ff:fe09:2739/64 scope link
>>>>        valid_lft forever preferred_lft forever
>>>> 403: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
>>>> fq_codel master virbr0 state UNKNOWN group default qlen 1000
>>>>     link/ether fe:54:00:ea:6b:18 brd ff:ff:ff:ff:ff:ff
>>>>     inet6 fe80::fc54:ff:feea:6b18/64 scope link
>>>>        valid_lft forever preferred_lft forever
>>>>
>>>
>>> I could not reproduce this locally by simply running netperf through 
>>> a mlx4 card. Some more questions:
>>>
>>> - What kind of workloads did you run in guest?
>>> - Did you meet this issue in a specific type of network card (I 
>>> guess broadcom is used in this case)?
>>> - Virbr0 looks like a bridge created by libvirt that did NAT and 
>>> other stuffs, can you still hit this issue if you don't use virbr0?
>>>
>>> And what's more important, zerocopy is known to have issues, for 
>>> production environment, need to disable it through vhost_net module 
>>> parameters.
>>>
>>> Thanks
>>
>> I'm deploying an overcloud through a undercloud virtual machine... 
>> The VM has 4vCPUs and 16GB of RAM as well as to virtio nics so I'm 
>> using only virtual hardware here.
>> I spawn 7 VMs on the hypervisor and deploy an overcloud using tripleo 
>> on them ... everything's virtual and if I remove the bridge, then 
>> I'll have to configure each VMs differently.
>> The load is quite high on the VM that won't shutdown but when I shut 
>> it down, it's doing nothing ...   This is a hard bug to troubleshoot 
>> and I can't bisect the kernel because at some
>> point the system simply won't boot properly.
>
> I've disabled zerocopy with the following:
>
> [root@zappa modprobe.d]# cat vhost-net.conf
> options vhost_net  experimental_zcopytx=0
>
>
> And I haven't reproduce this issue so far.   The problem I have right 
> now is that experimental_zcopytx has been enabled by default with this 
> commit:
>
> commit f9611c43ab0ddaf547b395c90fb842f55959334c
> Author: Michael S. Tsirkin <mst@redhat.com>
> Date:   Thu Dec 6 14:56:00 2012 +0200
>
>     vhost-net: enable zerocopy tx by default
>
>     Zero copy TX has been around for a while now.
>     We seem to be down to eliminating theoretical bugs
>     and performance tuning at this point:
>     it's probably time to enable it by default so that
>     most users get the benefit.
>
>     Keep the flag around meanwhile so users can experiment
>     with disabling this if they experience regressions.
>     I expect that we will remove it in the future.
>
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>
> I'll try some more pass in producing this issue and I'll keep you posted.
>
> Thank you very much,
>
> David Hill
>

Thanks. Zerocopy is disabled by several distribution by default. For 
upstream, the only reason to let it on is to hope more developers can 
help and fix the issues.

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Dave Hill]

On 11/28/2017 8:52 PM, Jason Wang wrote:
>
>
> On 2017年11月29日 02:00, David Hill wrote:
>>
>>
>> On 2017-11-27 02:38 PM, David Hill wrote:
>>>
>>>
>>> On 2017-11-26 10:44 PM, Jason Wang wrote:
>>>>
>>>>
>>>> On 2017年11月25日 00:22, David Hill wrote:
>>>>> The VMs all have 2 vNICs ... and this is the hypervisor:
>>>>>
>>>>> [root@zappa ~]# brctl show
>>>>> bridge name    bridge id        STP enabled    interfaces
>>>>> virbr0        8000.525400914858    yes        virbr0-nic
>>>>>                             vnet0
>>>>>                             vnet1
>>>>>
>>>>>
>>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state 
>>>>> UNKNOWN group default qlen 1000
>>>>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>>>>     inet 127.0.0.1/8 scope host lo
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet6 ::1/128 scope host
>>>>>        valid_lft forever preferred_lft forever
>>>>> 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
>>>>> UP group default qlen 1000
>>>>>     link/ether 84:2b:2b:13:f2:91 brd ff:ff:ff:ff:ff:ff
>>>>>     inet redacted/24 brd 173.178.138.255 scope global dynamic eno1
>>>>>        valid_lft 48749sec preferred_lft 48749sec
>>>>>     inet6 fe80::862b:2bff:fe13:f291/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
>>>>> UP group default qlen 1000
>>>>>     link/ether 84:2b:2b:13:f2:92 brd ff:ff:ff:ff:ff:ff
>>>>>     inet 192.168.1.3/24 brd 192.168.1.255 scope global eno2
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet6 fe80::862b:2bff:fe13:f292/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
>>>>> noqueue state UP group default qlen 1000
>>>>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>>>>>     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
>>>>>
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.10/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.11/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.12/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.15/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.16/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.17/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.18/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.31/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.32/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.33/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.34/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.35/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.36/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.37/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.45/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.46/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.47/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.48/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.49/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.50/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet 192.168.122.51/32 scope global virbr0
>>>>>        valid_lft forever preferred_lft forever
>>>>> 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel 
>>>>> master virbr0 state DOWN group default qlen 1000
>>>>>     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
>>>>> 125: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 
>>>>> qdisc fq_codel state UNKNOWN group default qlen 100
>>>>>     link/none
>>>>>     inet 10.10.122.28/21 brd 10.10.127.255 scope global tun0
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet6 fe80::1f9b:bfd4:e9c9:2059/64 scope link stable-privacy
>>>>>        valid_lft forever preferred_lft forever
>>>>> 402: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
>>>>> fq_codel master virbr0 state UNKNOWN group default qlen 1000
>>>>>     link/ether fe:54:00:09:27:39 brd ff:ff:ff:ff:ff:ff
>>>>>     inet6 fe80::fc54:ff:fe09:2739/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 403: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
>>>>> fq_codel master virbr0 state UNKNOWN group default qlen 1000
>>>>>     link/ether fe:54:00:ea:6b:18 brd ff:ff:ff:ff:ff:ff
>>>>>     inet6 fe80::fc54:ff:feea:6b18/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>>
>>>>
>>>> I could not reproduce this locally by simply running netperf 
>>>> through a mlx4 card. Some more questions:
>>>>
>>>> - What kind of workloads did you run in guest?
>>>> - Did you meet this issue in a specific type of network card (I 
>>>> guess broadcom is used in this case)?
>>>> - Virbr0 looks like a bridge created by libvirt that did NAT and 
>>>> other stuffs, can you still hit this issue if you don't use virbr0?
>>>>
>>>> And what's more important, zerocopy is known to have issues, for 
>>>> production environment, need to disable it through vhost_net module 
>>>> parameters.
>>>>
>>>> Thanks
>>>
>>> I'm deploying an overcloud through a undercloud virtual machine... 
>>> The VM has 4vCPUs and 16GB of RAM as well as to virtio nics so I'm 
>>> using only virtual hardware here.
>>> I spawn 7 VMs on the hypervisor and deploy an overcloud using 
>>> tripleo on them ... everything's virtual and if I remove the bridge, 
>>> then I'll have to configure each VMs differently.
>>> The load is quite high on the VM that won't shutdown but when I shut 
>>> it down, it's doing nothing ...   This is a hard bug to troubleshoot 
>>> and I can't bisect the kernel because at some
>>> point the system simply won't boot properly.
>>
>> I've disabled zerocopy with the following:
>>
>> [root@zappa modprobe.d]# cat vhost-net.conf
>> options vhost_net  experimental_zcopytx=0
>>
>>
>> And I haven't reproduce this issue so far.   The problem I have right 
>> now is that experimental_zcopytx has been enabled by default with 
>> this commit:
>>
>> commit f9611c43ab0ddaf547b395c90fb842f55959334c
>> Author: Michael S. Tsirkin <mst@redhat.com>
>> Date:   Thu Dec 6 14:56:00 2012 +0200
>>
>>     vhost-net: enable zerocopy tx by default
>>
>>     Zero copy TX has been around for a while now.
>>     We seem to be down to eliminating theoretical bugs
>>     and performance tuning at this point:
>>     it's probably time to enable it by default so that
>>     most users get the benefit.
>>
>>     Keep the flag around meanwhile so users can experiment
>>     with disabling this if they experience regressions.
>>     I expect that we will remove it in the future.
>>
>>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>
>> I'll try some more pass in producing this issue and I'll keep you 
>> posted.
>>
>> Thank you very much,
>>
>> David Hill
>>
>
> Thanks. Zerocopy is disabled by several distribution by default. For 
> upstream, the only reason to let it on is to hope more developers can 
> help and fix the issues.
>
>
So I never hit this issue with previous kernel and this issue started 
happening with the v4.14-rc series.   I'm using rawhide so perhaps this 
is why it isn't disabled by default but I have to mention it's an update 
of FC25 up to FC28 and it never got disabled.
Perhaps it should be disabled in Fedora too if it's not the case... I'm 
not sure this is the place to discuss this ... is it?

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年11月29日 10:52, Dave Hill wrote:
>>>
>>
>> Thanks. Zerocopy is disabled by several distribution by default. For 
>> upstream, the only reason to let it on is to hope more developers can 
>> help and fix the issues.
>>
>>
> So I never hit this issue with previous kernel and this issue started 
> happening with the v4.14-rc series.


Right, this still need to be investigated if it was introduced recently.

Looking at git history, the only suspected commit is for 4.14 is

commit 1e6f74536de08b5e50cf0e37e735911c2cef7c62
Author: Willem de Bruijn <willemb@google.com>
Date:   Fri Oct 6 13:22:31 2017 -0400

     vhost_net: do not stall on zerocopy depletion

Maybe you can try to revert it and see.

If it does not solve your issue, I suspect there's bug elsewhere that 
cause a packet to be held for very long time.

>   I'm using rawhide so perhaps this is why it isn't disabled by 
> default but I have to mention it's an update of FC25 up to FC28 and it 
> never got disabled.
> Perhaps it should be disabled in Fedora too if it's not the case... 
> I'm not sure this is the place to discuss this ... is it? 

Probably not, but I guess Fedora tries to use new technology aggressively.

Thanks

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-11-29 12:15 AM, Jason Wang wrote:
>
>
> On 2017年11月29日 10:52, Dave Hill wrote:
>>>>
>>>
>>> Thanks. Zerocopy is disabled by several distribution by default. For 
>>> upstream, the only reason to let it on is to hope more developers 
>>> can help and fix the issues.
>>>
>>>
>> So I never hit this issue with previous kernel and this issue started 
>> happening with the v4.14-rc series.
>
>
> Right, this still need to be investigated if it was introduced recently.
>
> Looking at git history, the only suspected commit is for 4.14 is
>
> commit 1e6f74536de08b5e50cf0e37e735911c2cef7c62
> Author: Willem de Bruijn <willemb@google.com>
> Date:   Fri Oct 6 13:22:31 2017 -0400
>
>     vhost_net: do not stall on zerocopy depletion
>
> Maybe you can try to revert it and see.
>
> If it does not solve your issue, I suspect there's bug elsewhere that 
> cause a packet to be held for very long time.
>
>>   I'm using rawhide so perhaps this is why it isn't disabled by 
>> default but I have to mention it's an update of FC25 up to FC28 and 
>> it never got disabled.
>> Perhaps it should be disabled in Fedora too if it's not the case... 
>> I'm not sure this is the place to discuss this ... is it? 
>
> Probably not, but I guess Fedora tries to use new technology 
> aggressively.
>
> Thanks

I can revert that commit in 4.15-rc1 but I can't find it in 4.14.2 ...  
Is there another commit that could affect this ?

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年11月30日 03:13, David Hill wrote:
>
>
> On 2017-11-29 12:15 AM, Jason Wang wrote:
>>
>>
>> On 2017年11月29日 10:52, Dave Hill wrote:
>>>>>
>>>>
>>>> Thanks. Zerocopy is disabled by several distribution by default. 
>>>> For upstream, the only reason to let it on is to hope more 
>>>> developers can help and fix the issues.
>>>>
>>>>
>>> So I never hit this issue with previous kernel and this issue 
>>> started happening with the v4.14-rc series.
>>
>>
>> Right, this still need to be investigated if it was introduced recently.
>>
>> Looking at git history, the only suspected commit is for 4.14 is
>>
>> commit 1e6f74536de08b5e50cf0e37e735911c2cef7c62
>> Author: Willem de Bruijn <willemb@google.com>
>> Date:   Fri Oct 6 13:22:31 2017 -0400
>>
>>     vhost_net: do not stall on zerocopy depletion
>>
>> Maybe you can try to revert it and see.
>>
>> If it does not solve your issue, I suspect there's bug elsewhere that 
>> cause a packet to be held for very long time.
>>
>>>   I'm using rawhide so perhaps this is why it isn't disabled by 
>>> default but I have to mention it's an update of FC25 up to FC28 and 
>>> it never got disabled.
>>> Perhaps it should be disabled in Fedora too if it's not the case... 
>>> I'm not sure this is the place to discuss this ... is it? 
>>
>> Probably not, but I guess Fedora tries to use new technology 
>> aggressively.
>>
>> Thanks
>
> I can revert that commit in 4.15-rc1 but I can't find it in 4.14.2 
> ...  Is there another commit that could affect this ?

My bad, the suspicious is then:

1f8b977ab32dc5d148f103326e80d9097f1cefb5 ("sock: enable MSG_ZEROCOPY")
c1d1b437816f0afa99202be3cb650c9d174667bc ("net: convert (struct 
ubuf_info)->refcnt to refcount_t")

Thanks

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-11-29 09:42 PM, Jason Wang wrote:
>
>
> On 2017年11月30日 03:13, David Hill wrote:
>>
>>
>> On 2017-11-29 12:15 AM, Jason Wang wrote:
>>>
>>>
>>> On 2017年11月29日 10:52, Dave Hill wrote:
>>>>>>
>>>>>
>>>>> Thanks. Zerocopy is disabled by several distribution by default. 
>>>>> For upstream, the only reason to let it on is to hope more 
>>>>> developers can help and fix the issues.
>>>>>
>>>>>
>>>> So I never hit this issue with previous kernel and this issue 
>>>> started happening with the v4.14-rc series.
>>>
>>>
>>> Right, this still need to be investigated if it was introduced 
>>> recently.
>>>
>>> Looking at git history, the only suspected commit is for 4.14 is
>>>
>>> commit 1e6f74536de08b5e50cf0e37e735911c2cef7c62
>>> Author: Willem de Bruijn <willemb@google.com>
>>> Date:   Fri Oct 6 13:22:31 2017 -0400
>>>
>>>     vhost_net: do not stall on zerocopy depletion
>>>
>>> Maybe you can try to revert it and see.
>>>
>>> If it does not solve your issue, I suspect there's bug elsewhere 
>>> that cause a packet to be held for very long time.
>>>
>>>>   I'm using rawhide so perhaps this is why it isn't disabled by 
>>>> default but I have to mention it's an update of FC25 up to FC28 and 
>>>> it never got disabled.
>>>> Perhaps it should be disabled in Fedora too if it's not the case... 
>>>> I'm not sure this is the place to discuss this ... is it? 
>>>
>>> Probably not, but I guess Fedora tries to use new technology 
>>> aggressively.
>>>
>>> Thanks
>>
>> I can revert that commit in 4.15-rc1 but I can't find it in 4.14.2 
>> ...  Is there another commit that could affect this ?
>
> My bad, the suspicious is then:
>
> 1f8b977ab32dc5d148f103326e80d9097f1cefb5 ("sock: enable MSG_ZEROCOPY")
> c1d1b437816f0afa99202be3cb650c9d174667bc ("net: convert (struct 
> ubuf_info)->refcnt to refcount_t")
>
> Thanks
>

Reverting those two commits breaks kernel compilation:

net/core/dev.c: In function ‘dev_queue_xmit_nit’:
net/core/dev.c:1952:8: error: implicit declaration of function 
‘skb_orphan_frags_rx’; did you mean ‘skb_orphan_frags’? 
[-Werror=implicit-function-declaration]
    if (!skb_orphan_frags_rx(skb2, GFP_ATOMIC))
         ^~~~~~~~~~~~~~~~~~~
         skb_orphan_frags


I changed skb_orphan_frags_rx to skb_orphan_frags and it compiled but 
will everything blow up?

Thanks,
Dave

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-11-30 03:52 PM, David Hill wrote:
>
>
> On 2017-11-29 09:42 PM, Jason Wang wrote:
>>
>>
>> On 2017年11月30日 03:13, David Hill wrote:
>>>
>>>
>>> On 2017-11-29 12:15 AM, Jason Wang wrote:
>>>>
>>>>
>>>> On 2017年11月29日 10:52, Dave Hill wrote:
>>>>>>>
>>>>>>
>>>>>> Thanks. Zerocopy is disabled by several distribution by default. 
>>>>>> For upstream, the only reason to let it on is to hope more 
>>>>>> developers can help and fix the issues.
>>>>>>
>>>>>>
>>>>> So I never hit this issue with previous kernel and this issue 
>>>>> started happening with the v4.14-rc series.
>>>>
>>>>
>>>> Right, this still need to be investigated if it was introduced 
>>>> recently.
>>>>
>>>> Looking at git history, the only suspected commit is for 4.14 is
>>>>
>>>> commit 1e6f74536de08b5e50cf0e37e735911c2cef7c62
>>>> Author: Willem de Bruijn <willemb@google.com>
>>>> Date:   Fri Oct 6 13:22:31 2017 -0400
>>>>
>>>>     vhost_net: do not stall on zerocopy depletion
>>>>
>>>> Maybe you can try to revert it and see.
>>>>
>>>> If it does not solve your issue, I suspect there's bug elsewhere 
>>>> that cause a packet to be held for very long time.
>>>>
>>>>>   I'm using rawhide so perhaps this is why it isn't disabled by 
>>>>> default but I have to mention it's an update of FC25 up to FC28 
>>>>> and it never got disabled.
>>>>> Perhaps it should be disabled in Fedora too if it's not the 
>>>>> case... I'm not sure this is the place to discuss this ... is it? 
>>>>
>>>> Probably not, but I guess Fedora tries to use new technology 
>>>> aggressively.
>>>>
>>>> Thanks
>>>
>>> I can revert that commit in 4.15-rc1 but I can't find it in 4.14.2 
>>> ...  Is there another commit that could affect this ?
>>
>> My bad, the suspicious is then:
>>
>> 1f8b977ab32dc5d148f103326e80d9097f1cefb5 ("sock: enable MSG_ZEROCOPY")
>> c1d1b437816f0afa99202be3cb650c9d174667bc ("net: convert (struct 
>> ubuf_info)->refcnt to refcount_t")
>>
>> Thanks
>>
>
> Reverting those two commits breaks kernel compilation:
>
> net/core/dev.c: In function ‘dev_queue_xmit_nit’:
> net/core/dev.c:1952:8: error: implicit declaration of function 
> ‘skb_orphan_frags_rx’; did you mean ‘skb_orphan_frags’? 
> [-Werror=implicit-function-declaration]
>    if (!skb_orphan_frags_rx(skb2, GFP_ATOMIC))
>         ^~~~~~~~~~~~~~~~~~~
>         skb_orphan_frags
>
>
> I changed skb_orphan_frags_rx to skb_orphan_frags and it compiled but 
> will everything blow up?
>
> Thanks,
> Dave

Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too ... 
compiling and I'll keep you posted.

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-11-30 03:59 PM, David Hill wrote:
>
>
> On 2017-11-30 03:52 PM, David Hill wrote:
>>
>>
>> On 2017-11-29 09:42 PM, Jason Wang wrote:
>>>
>>>
>>> On 2017年11月30日 03:13, David Hill wrote:
>>>>
>>>>
>>>> On 2017-11-29 12:15 AM, Jason Wang wrote:
>>>>>
>>>>>
>>>>> On 2017年11月29日 10:52, Dave Hill wrote:
>>>>>>>>
>>>>>>>
>>>>>>> Thanks. Zerocopy is disabled by several distribution by default. 
>>>>>>> For upstream, the only reason to let it on is to hope more 
>>>>>>> developers can help and fix the issues.
>>>>>>>
>>>>>>>
>>>>>> So I never hit this issue with previous kernel and this issue 
>>>>>> started happening with the v4.14-rc series.
>>>>>
>>>>>
>>>>> Right, this still need to be investigated if it was introduced 
>>>>> recently.
>>>>>
>>>>> Looking at git history, the only suspected commit is for 4.14 is
>>>>>
>>>>> commit 1e6f74536de08b5e50cf0e37e735911c2cef7c62
>>>>> Author: Willem de Bruijn <willemb@google.com>
>>>>> Date:   Fri Oct 6 13:22:31 2017 -0400
>>>>>
>>>>>     vhost_net: do not stall on zerocopy depletion
>>>>>
>>>>> Maybe you can try to revert it and see.
>>>>>
>>>>> If it does not solve your issue, I suspect there's bug elsewhere 
>>>>> that cause a packet to be held for very long time.
>>>>>
>>>>>>   I'm using rawhide so perhaps this is why it isn't disabled by 
>>>>>> default but I have to mention it's an update of FC25 up to FC28 
>>>>>> and it never got disabled.
>>>>>> Perhaps it should be disabled in Fedora too if it's not the 
>>>>>> case... I'm not sure this is the place to discuss this ... is it? 
>>>>>
>>>>> Probably not, but I guess Fedora tries to use new technology 
>>>>> aggressively.
>>>>>
>>>>> Thanks
>>>>
>>>> I can revert that commit in 4.15-rc1 but I can't find it in 4.14.2 
>>>> ...  Is there another commit that could affect this ?
>>>
>>> My bad, the suspicious is then:
>>>
>>> 1f8b977ab32dc5d148f103326e80d9097f1cefb5 ("sock: enable MSG_ZEROCOPY")
>>> c1d1b437816f0afa99202be3cb650c9d174667bc ("net: convert (struct 
>>> ubuf_info)->refcnt to refcount_t")
>>>
>>> Thanks
>>>
>>
>> Reverting those two commits breaks kernel compilation:
>>
>> net/core/dev.c: In function ‘dev_queue_xmit_nit’:
>> net/core/dev.c:1952:8: error: implicit declaration of function 
>> ‘skb_orphan_frags_rx’; did you mean ‘skb_orphan_frags’? 
>> [-Werror=implicit-function-declaration]
>>    if (!skb_orphan_frags_rx(skb2, GFP_ATOMIC))
>>         ^~~~~~~~~~~~~~~~~~~
>>         skb_orphan_frags
>>
>>
>> I changed skb_orphan_frags_rx to skb_orphan_frags and it compiled but 
>> will everything blow up?
>>
>> Thanks,
>> Dave
>
> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too ... 
> compiling and I'll keep you posted.

So I'm still able to reproduce this issue even with reverting these 3 
commits.  Would you have other suspect commits ?

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Harald Moeller]

Hello, my name is Harry and this is my first post here, hope I'm doing 
this the right way, sorry if not ...

I'm not a subscriber to the full list yet so I understand I shall ask 
you to be personally CCed.

I am following this as I do experience the same (or sort-a same) issue 
with 4.14.2.

My setup is more simple, just an oVirt host shutting down some VMs. 
Doesn't happen all the time but I'd say around 3 from 10.

This is what I see (slightly different from David):

Dec 01 16:11:53 oVirtHost01.xyz.net kernel: INFO: task qemu-kvm:1173 
blocked for more than 120 seconds.
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:       Tainted: G          
I     4.14.2-1.el7.hakimo.x86_64 #4
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: qemu-kvm        D 0  
1173      1 0x00000084
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: Call Trace:
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: __schedule+0x28d/0x880
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  schedule+0x36/0x80
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: 
vhost_net_ubuf_put_and_wait+0x61/0x90 [vhost_net]
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  ? remove_wait_queue+0x60/0x60
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: vhost_net_ioctl+0x317/0x8e0 
[vhost_net]
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: do_vfs_ioctl+0xa7/0x5f0
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  SyS_ioctl+0x79/0x90
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: do_syscall_64+0x67/0x1b0
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: 
entry_SYSCALL64_slow_path+0x25/0x25
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RIP: 0033:0x7fb8862d1107
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RSP: 002b:00007fff4acd7e58 
EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RAX: ffffffffffffffda RBX: 
000055abaa2d29c0 RCX: 00007fb8862d1107
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RDX: 00007fff4acd7e60 RSI: 
000000004008af30 RDI: 0000000000000028
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RBP: 00007fff4acd7e60 R08: 
000055aba805e10f R09: 00000000ffffffff
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: R10: 0000000000000004 R11: 
0000000000000246 R12: 000055ababf32510
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: R13: 0000000000000001 R14: 
000055ababf32498 R15: 000055abaa2a0b40

This is still happening after reverting the three suggested commits

1f8b977ab32dc5d148f103326e80d9097f1cefb5 ("sock: enable MSG_ZEROCOPY")

c1d1b437816f0afa99202be3cb650c9d174667bc ("net: convert (struct 
ubuf_info)->refcnt to refcount_t")

581fe0ea61584d88072527ae9fb9dcb9d1f2783e {"net: orphan frags on 
stand-alone ptype in dev_queue_xmit_nit"}

Anything I could be helpful with trying to solve this? Any more info I 
could provide?

Harry

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Harald Moeller]

Hello, my name is Harry and this is my first post here, hope I'm doing 
this the right way, sorry if not ...

I'm not a subscriber to the full list yet so I understand I shall ask 
you to be personally CCed.

I am following this as I do experience the same (or sort-a same) issue 
with 4.14.2.

My setup is more simple, just an oVirt host shutting down some VMs. 
Doesn't happen all the time but I'd say around 3 from 10.

This is what I see (slightly different from David):

Dec 01 16:11:53 oVirtHost01.xyz.net kernel: INFO: task qemu-kvm:1173 
blocked for more than 120 seconds.
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:       Tainted: G          
I     4.14.2-1.el7.hakimo.x86_64 #4
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: qemu-kvm        D 0 
1173      1 0x00000084
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: Call Trace:
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: __schedule+0x28d/0x880
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  schedule+0x36/0x80
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: 
vhost_net_ubuf_put_and_wait+0x61/0x90 [vhost_net]
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  ? remove_wait_queue+0x60/0x60
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: vhost_net_ioctl+0x317/0x8e0 
[vhost_net]
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: do_vfs_ioctl+0xa7/0x5f0
Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  SyS_ioctl+0x79/0x90
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: do_syscall_64+0x67/0x1b0
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: 
entry_SYSCALL64_slow_path+0x25/0x25
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RIP: 0033:0x7fb8862d1107
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RSP: 002b:00007fff4acd7e58 
EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RAX: ffffffffffffffda RBX: 
000055abaa2d29c0 RCX: 00007fb8862d1107
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RDX: 00007fff4acd7e60 RSI: 
000000004008af30 RDI: 0000000000000028
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RBP: 00007fff4acd7e60 R08: 
000055aba805e10f R09: 00000000ffffffff
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: R10: 0000000000000004 R11: 
0000000000000246 R12: 000055ababf32510
Dec 01 16:11:53 oVirtHost01.xyz.net kernel: R13: 0000000000000001 R14: 
000055ababf32498 R15: 000055abaa2a0b40

This is still happening after reverting the three suggested commits

1f8b977ab32dc5d148f103326e80d9097f1cefb5 ("sock: enable MSG_ZEROCOPY")

c1d1b437816f0afa99202be3cb650c9d174667bc ("net: convert (struct 
ubuf_info)->refcnt to refcount_t")

581fe0ea61584d88072527ae9fb9dcb9d1f2783e {"net: orphan frags on 
stand-alone ptype in dev_queue_xmit_nit"}

Anything I could be helpful with trying to solve this? Any more info I 
could provide?

Harry

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年12月02日 00:38, David Hill wrote:
>>
>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too ... 
>> compiling and I'll keep you posted.
>
> So I'm still able to reproduce this issue even with reverting these 3 
> commits.  Would you have other suspect commits ? 

Thanks for the testing. No, I don't have other suspect commits.

Looks like somebody else it hitting your issue too (see 
https://www.spinics.net/lists/netdev/msg468319.html)

But he claims the issue were fixed by using qemu 2.10.1.

So you may:

-try to see if qemu 2.10.1 solves your issue
-if not, try to see if commit 2ddf71e23cc246e95af72a6deed67b4a50a7b81c 
("net: add notifier hooks for devmap bpf map") is the first bad commit
-if not, maybe you can continue your bisection through git bisect skip

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-12-03 11:08 PM, Jason Wang wrote:
>
>
> On 2017年12月02日 00:38, David Hill wrote:
>>>
>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too ... 
>>> compiling and I'll keep you posted.
>>
>> So I'm still able to reproduce this issue even with reverting these 3 
>> commits.  Would you have other suspect commits ? 
>
> Thanks for the testing. No, I don't have other suspect commits.
>
> Looks like somebody else it hitting your issue too (see 
> https://www.spinics.net/lists/netdev/msg468319.html)
>
> But he claims the issue were fixed by using qemu 2.10.1.
>
> So you may:
>
> -try to see if qemu 2.10.1 solves your issue
It didn't solve it for him... it's only harder to reproduce. [1]
> -if not, try to see if commit 2ddf71e23cc246e95af72a6deed67b4a50a7b81c 
> ("net: add notifier hooks for devmap bpf map") is the first bad commit
I'll try to see what I can do here
> -if not, maybe you can continue your bisection through git bisect skip
>
Some commits are so broken that the system won't boot ...  What I fear 
is that if I git bisect skip those commits, I'll also skip the commit 
culprit of my original problem

[1] https://www.spinics.net/lists/netdev/msg469887.html

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

Have you tried adding this:


cat<<EOF>/etc/modprobe.d/vhost-net.conf
options vhost_net  experimental_zcopytx=0
EOF

reboot


Other than this, you can try bisecting but in my case, the system wont 
boot when reaching a given commit.



On 2017-12-02 11:37 AM, Harald Moeller wrote:
> Hello, my name is Harry and this is my first post here, hope I'm doing 
> this the right way, sorry if not ...
>
> I'm not a subscriber to the full list yet so I understand I shall ask 
> you to be personally CCed.
>
> I am following this as I do experience the same (or sort-a same) issue 
> with 4.14.2.
>
> My setup is more simple, just an oVirt host shutting down some VMs. 
> Doesn't happen all the time but I'd say around 3 from 10.
>
> This is what I see (slightly different from David):
>
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: INFO: task qemu-kvm:1173 
> blocked for more than 120 seconds.
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel:       Tainted: G          
> I     4.14.2-1.el7.hakimo.x86_64 #4
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: "echo 0 > 
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: qemu-kvm        D 0 
> 1173      1 0x00000084
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: Call Trace:
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: __schedule+0x28d/0x880
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  schedule+0x36/0x80
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: 
> vhost_net_ubuf_put_and_wait+0x61/0x90 [vhost_net]
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  ? 
> remove_wait_queue+0x60/0x60
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: 
> vhost_net_ioctl+0x317/0x8e0 [vhost_net]
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: do_vfs_ioctl+0xa7/0x5f0
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel:  SyS_ioctl+0x79/0x90
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: do_syscall_64+0x67/0x1b0
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: 
> entry_SYSCALL64_slow_path+0x25/0x25
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RIP: 0033:0x7fb8862d1107
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RSP: 002b:00007fff4acd7e58 
> EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RAX: ffffffffffffffda RBX: 
> 000055abaa2d29c0 RCX: 00007fb8862d1107
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RDX: 00007fff4acd7e60 RSI: 
> 000000004008af30 RDI: 0000000000000028
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: RBP: 00007fff4acd7e60 R08: 
> 000055aba805e10f R09: 00000000ffffffff
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: R10: 0000000000000004 R11: 
> 0000000000000246 R12: 000055ababf32510
> Dec 01 16:11:53 oVirtHost01.xyz.net kernel: R13: 0000000000000001 R14: 
> 000055ababf32498 R15: 000055abaa2a0b40
>
> This is still happening after reverting the three suggested commits
>
> 1f8b977ab32dc5d148f103326e80d9097f1cefb5 ("sock: enable MSG_ZEROCOPY")
>
> c1d1b437816f0afa99202be3cb650c9d174667bc ("net: convert (struct 
> ubuf_info)->refcnt to refcount_t")
>
> 581fe0ea61584d88072527ae9fb9dcb9d1f2783e {"net: orphan frags on 
> stand-alone ptype in dev_queue_xmit_nit"}
>
> Anything I could be helpful with trying to solve this? Any more info I 
> could provide?
>
> Harry
>

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-12-04 02:51 PM, David Hill wrote:
>
> On 2017-12-03 11:08 PM, Jason Wang wrote:
>>
>>
>> On 2017年12月02日 00:38, David Hill wrote:
>>>>
>>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too 
>>>> ... compiling and I'll keep you posted.
>>>
>>> So I'm still able to reproduce this issue even with reverting these 
>>> 3 commits.  Would you have other suspect commits ? 
>>
>> Thanks for the testing. No, I don't have other suspect commits.
>>
>> Looks like somebody else it hitting your issue too (see 
>> https://www.spinics.net/lists/netdev/msg468319.html)
>>
>> But he claims the issue were fixed by using qemu 2.10.1.
>>
>> So you may:
>>
>> -try to see if qemu 2.10.1 solves your issue
> It didn't solve it for him... it's only harder to reproduce. [1]
>> -if not, try to see if commit 
>> 2ddf71e23cc246e95af72a6deed67b4a50a7b81c ("net: add notifier hooks 
>> for devmap bpf map") is the first bad commit
> I'll try to see what I can do here
I'm looking at that commit and it's been introduced before v4.13 if I'm 
not mistaken while this issue appeared between v4.13 and v4.14-rc1 .  
Between those two releases, there're  1352 commits.
Is there a way to quickly know which commits are touching vhost-net, 
zerocopy ?


[ 7496.553044]  __schedule+0x2dc/0xbb0
[ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
[ 7496.553074]  schedule+0x3d/0x90
[ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
[ 7496.553100]  ? finish_wait+0x90/0x90
[ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
[ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
[ 7496.553166]  SyS_ioctl+0x79/0x90
[ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe

>> -if not, maybe you can continue your bisection through git bisect skip
>>
> Some commits are so broken that the system won't boot ...  What I fear 
> is that if I git bisect skip those commits, I'll also skip the commit 
> culprit of my original problem
>
> [1] https://www.spinics.net/lists/netdev/msg469887.html

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-12-06 11:34 PM, David Hill wrote:
>
>
> On 2017-12-04 02:51 PM, David Hill wrote:
>>
>> On 2017-12-03 11:08 PM, Jason Wang wrote:
>>>
>>>
>>> On 2017年12月02日 00:38, David Hill wrote:
>>>>>
>>>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too 
>>>>> ... compiling and I'll keep you posted.
>>>>
>>>> So I'm still able to reproduce this issue even with reverting these 
>>>> 3 commits.  Would you have other suspect commits ? 
>>>
>>> Thanks for the testing. No, I don't have other suspect commits.
>>>
>>> Looks like somebody else it hitting your issue too (see 
>>> https://www.spinics.net/lists/netdev/msg468319.html)
>>>
>>> But he claims the issue were fixed by using qemu 2.10.1.
>>>
>>> So you may:
>>>
>>> -try to see if qemu 2.10.1 solves your issue
>> It didn't solve it for him... it's only harder to reproduce. [1]
>>> -if not, try to see if commit 
>>> 2ddf71e23cc246e95af72a6deed67b4a50a7b81c ("net: add notifier hooks 
>>> for devmap bpf map") is the first bad commit
>> I'll try to see what I can do here
> I'm looking at that commit and it's been introduced before v4.13 if 
> I'm not mistaken while this issue appeared between v4.13 and v4.14-rc1 
> .  Between those two releases, there're  1352 commits.
> Is there a way to quickly know which commits are touching vhost-net, 
> zerocopy ?
>
>
> [ 7496.553044]  __schedule+0x2dc/0xbb0
> [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
> [ 7496.553074]  schedule+0x3d/0x90
> [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
> [ 7496.553100]  ? finish_wait+0x90/0x90
> [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
> [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
> [ 7496.553166]  SyS_ioctl+0x79/0x90
> [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe

That vhost_net_ubuf_put_and)wait call has been changed in this commit 
with the following comment:

commit 0ad8b480d6ee916aa84324f69acf690142aecd0e
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Thu Feb 13 11:42:05 2014 +0200

     vhost: fix ref cnt checking deadlock

     vhost checked the counter within the refcnt before decrementing.  It
     really wanted to know that it is the one that has the last 
reference, as
     a way to batch freeing resources a bit more efficiently.

     Note: we only let refcount go to 0 on device release.

     This works well but we now access the ref counter twice so there's a
     race: all users might see a high count and decide to defer freeing
     resources.
     In the end no one initiates freeing resources until the last reference
     is gone (which is on VM shotdown so might happen after a looooong 
time).

     Let's do what we probably should have done straight away:
     switch from kref to plain atomic, documenting the
     semantics, return the refcount value atomically after decrement,
     then use that to avoid the deadlock.

     Reported-by: Qin Chuanyu <qinchuanyu@huawei.com>
     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
     Acked-by: Jason Wang <jasowang@redhat.com>
     Signed-off-by: David S. Miller <davem@davemloft.net>



So at this point, are we hitting a deadlock when using 
experimental_zcopytx ?


>
>>> -if not, maybe you can continue your bisection through git bisect skip
>>>
>> Some commits are so broken that the system won't boot ...  What I 
>> fear is that if I git bisect skip those commits, I'll also skip the 
>> commit culprit of my original problem
>>
>> [1] https://www.spinics.net/lists/netdev/msg469887.html
>

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年12月07日 12:34, David Hill wrote:
>
>
> On 2017-12-04 02:51 PM, David Hill wrote:
>>
>> On 2017-12-03 11:08 PM, Jason Wang wrote:
>>>
>>>
>>> On 2017年12月02日 00:38, David Hill wrote:
>>>>>
>>>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too 
>>>>> ... compiling and I'll keep you posted.
>>>>
>>>> So I'm still able to reproduce this issue even with reverting these 
>>>> 3 commits.  Would you have other suspect commits ? 
>>>
>>> Thanks for the testing. No, I don't have other suspect commits.
>>>
>>> Looks like somebody else it hitting your issue too (see 
>>> https://www.spinics.net/lists/netdev/msg468319.html)
>>>
>>> But he claims the issue were fixed by using qemu 2.10.1.
>>>
>>> So you may:
>>>
>>> -try to see if qemu 2.10.1 solves your issue
>> It didn't solve it for him... it's only harder to reproduce. [1]
>>> -if not, try to see if commit 
>>> 2ddf71e23cc246e95af72a6deed67b4a50a7b81c ("net: add notifier hooks 
>>> for devmap bpf map") is the first bad commit
>> I'll try to see what I can do here
> I'm looking at that commit and it's been introduced before v4.13 if 
> I'm not mistaken while this issue appeared between v4.13 and v4.14-rc1 
> .  Between those two releases, there're  1352 commits.
> Is there a way to quickly know which commits are touching vhost-net, 
> zerocopy ?
>
>
> [ 7496.553044]  __schedule+0x2dc/0xbb0
> [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
> [ 7496.553074]  schedule+0x3d/0x90
> [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
> [ 7496.553100]  ? finish_wait+0x90/0x90
> [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
> [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
> [ 7496.553166]  SyS_ioctl+0x79/0x90
> [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe

e.g you can do

#git log --oneline v4.13..v4.14-rc1 drivers/vhost/net.c
8b949be vhost_net: correctly check tx avail during rx busy polling
c1d1b43 net: convert (struct ubuf_info)->refcnt to refcount_t
1f8b977 sock: enable MSG_ZEROCOPY
7a68ada Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net

If I understand it correctly, you can still hit the issue before 
1f8b977? If yes, you probably can bisect between 7a68ada and 1f8b977.

Thanks

>
>>> -if not, maybe you can continue your bisection through git bisect skip
>>>
>> Some commits are so broken that the system won't boot ...  What I 
>> fear is that if I git bisect skip those commits, I'll also skip the 
>> commit culprit of my original problem
>>
>> [1] https://www.spinics.net/lists/netdev/msg469887.html
>

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年12月07日 12:42, David Hill wrote:
>
>
> On 2017-12-06 11:34 PM, David Hill wrote:
>>
>>
>> On 2017-12-04 02:51 PM, David Hill wrote:
>>>
>>> On 2017-12-03 11:08 PM, Jason Wang wrote:
>>>>
>>>>
>>>> On 2017年12月02日 00:38, David Hill wrote:
>>>>>>
>>>>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too 
>>>>>> ... compiling and I'll keep you posted.
>>>>>
>>>>> So I'm still able to reproduce this issue even with reverting 
>>>>> these 3 commits.  Would you have other suspect commits ? 
>>>>
>>>> Thanks for the testing. No, I don't have other suspect commits.
>>>>
>>>> Looks like somebody else it hitting your issue too (see 
>>>> https://www.spinics.net/lists/netdev/msg468319.html)
>>>>
>>>> But he claims the issue were fixed by using qemu 2.10.1.
>>>>
>>>> So you may:
>>>>
>>>> -try to see if qemu 2.10.1 solves your issue
>>> It didn't solve it for him... it's only harder to reproduce. [1]
>>>> -if not, try to see if commit 
>>>> 2ddf71e23cc246e95af72a6deed67b4a50a7b81c ("net: add notifier hooks 
>>>> for devmap bpf map") is the first bad commit
>>> I'll try to see what I can do here
>> I'm looking at that commit and it's been introduced before v4.13 if 
>> I'm not mistaken while this issue appeared between v4.13 and 
>> v4.14-rc1 .  Between those two releases, there're  1352 commits.
>> Is there a way to quickly know which commits are touching vhost-net, 
>> zerocopy ?
>>
>>
>> [ 7496.553044]  __schedule+0x2dc/0xbb0
>> [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>> [ 7496.553074]  schedule+0x3d/0x90
>> [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>> [ 7496.553100]  ? finish_wait+0x90/0x90
>> [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>> [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>> [ 7496.553166]  SyS_ioctl+0x79/0x90
>> [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>
> That vhost_net_ubuf_put_and)wait call has been changed in this commit 
> with the following comment:
>
> commit 0ad8b480d6ee916aa84324f69acf690142aecd0e
> Author: Michael S. Tsirkin <mst@redhat.com>
> Date:   Thu Feb 13 11:42:05 2014 +0200
>
>     vhost: fix ref cnt checking deadlock
>
>     vhost checked the counter within the refcnt before decrementing.  It
>     really wanted to know that it is the one that has the last 
> reference, as
>     a way to batch freeing resources a bit more efficiently.
>
>     Note: we only let refcount go to 0 on device release.
>
>     This works well but we now access the ref counter twice so there's a
>     race: all users might see a high count and decide to defer freeing
>     resources.
>     In the end no one initiates freeing resources until the last 
> reference
>     is gone (which is on VM shotdown so might happen after a looooong 
> time).
>
>     Let's do what we probably should have done straight away:
>     switch from kref to plain atomic, documenting the
>     semantics, return the refcount value atomically after decrement,
>     then use that to avoid the deadlock.
>
>     Reported-by: Qin Chuanyu <qinchuanyu@huawei.com>
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>     Acked-by: Jason Wang <jasowang@redhat.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
>
>
>
> So at this point, are we hitting a deadlock when using 
> experimental_zcopytx ? 

Yes. But there could be another possibility that it was not caused by 
vhost_net itself but other places that holds a packet.

Thanks

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-12-07 12:13 AM, Jason Wang wrote:
>
>
> On 2017年12月07日 12:42, David Hill wrote:
>>
>>
>> On 2017-12-06 11:34 PM, David Hill wrote:
>>>
>>>
>>> On 2017-12-04 02:51 PM, David Hill wrote:
>>>>
>>>> On 2017-12-03 11:08 PM, Jason Wang wrote:
>>>>>
>>>>>
>>>>> On 2017年12月02日 00:38, David Hill wrote:
>>>>>>>
>>>>>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e too 
>>>>>>> ... compiling and I'll keep you posted.
>>>>>>
>>>>>> So I'm still able to reproduce this issue even with reverting 
>>>>>> these 3 commits.  Would you have other suspect commits ? 
>>>>>
>>>>> Thanks for the testing. No, I don't have other suspect commits.
>>>>>
>>>>> Looks like somebody else it hitting your issue too (see 
>>>>> https://www.spinics.net/lists/netdev/msg468319.html)
>>>>>
>>>>> But he claims the issue were fixed by using qemu 2.10.1.
>>>>>
>>>>> So you may:
>>>>>
>>>>> -try to see if qemu 2.10.1 solves your issue
>>>> It didn't solve it for him... it's only harder to reproduce. [1]
>>>>> -if not, try to see if commit 
>>>>> 2ddf71e23cc246e95af72a6deed67b4a50a7b81c ("net: add notifier hooks 
>>>>> for devmap bpf map") is the first bad commit
>>>> I'll try to see what I can do here
>>> I'm looking at that commit and it's been introduced before v4.13 if 
>>> I'm not mistaken while this issue appeared between v4.13 and 
>>> v4.14-rc1 .  Between those two releases, there're 1352 commits.
>>> Is there a way to quickly know which commits are touching vhost-net, 
>>> zerocopy ?
>>>
>>>
>>> [ 7496.553044]  __schedule+0x2dc/0xbb0
>>> [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>>> [ 7496.553074]  schedule+0x3d/0x90
>>> [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>>> [ 7496.553100]  ? finish_wait+0x90/0x90
>>> [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>>> [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>>> [ 7496.553166]  SyS_ioctl+0x79/0x90
>>> [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>>
>> That vhost_net_ubuf_put_and)wait call has been changed in this commit 
>> with the following comment:
>>
>> commit 0ad8b480d6ee916aa84324f69acf690142aecd0e
>> Author: Michael S. Tsirkin <mst@redhat.com>
>> Date:   Thu Feb 13 11:42:05 2014 +0200
>>
>>     vhost: fix ref cnt checking deadlock
>>
>>     vhost checked the counter within the refcnt before decrementing.  It
>>     really wanted to know that it is the one that has the last 
>> reference, as
>>     a way to batch freeing resources a bit more efficiently.
>>
>>     Note: we only let refcount go to 0 on device release.
>>
>>     This works well but we now access the ref counter twice so there's a
>>     race: all users might see a high count and decide to defer freeing
>>     resources.
>>     In the end no one initiates freeing resources until the last 
>> reference
>>     is gone (which is on VM shotdown so might happen after a looooong 
>> time).
>>
>>     Let's do what we probably should have done straight away:
>>     switch from kref to plain atomic, documenting the
>>     semantics, return the refcount value atomically after decrement,
>>     then use that to avoid the deadlock.
>>
>>     Reported-by: Qin Chuanyu <qinchuanyu@huawei.com>
>>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>     Acked-by: Jason Wang <jasowang@redhat.com>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>>
>>
>>
>> So at this point, are we hitting a deadlock when using 
>> experimental_zcopytx ? 
>
> Yes. But there could be another possibility that it was not caused by 
> vhost_net itself but other places that holds a packet.
>
> Thanks

While bisecting, when I reach this commit 
46d4b68f891bee5d83a32508bfbd9778be6b1b63, the system kernel panic when I 
run virt-customize :

Message from syslogd@zappa at Dec  8 12:52:06 ...
  kernel:[  350.016376] Kernel panic - not syncing: Fatal exception in 
interrupt

I marked that commit as bad again.   Will continue bisecting!

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[David Hill]

On 2017-12-08 01:03 PM, David Hill wrote:
>
>
> On 2017-12-07 12:13 AM, Jason Wang wrote:
>>
>>
>> On 2017年12月07日 12:42, David Hill wrote:
>>>
>>>
>>> On 2017-12-06 11:34 PM, David Hill wrote:
>>>>
>>>>
>>>> On 2017-12-04 02:51 PM, David Hill wrote:
>>>>>
>>>>> On 2017-12-03 11:08 PM, Jason Wang wrote:
>>>>>>
>>>>>>
>>>>>> On 2017年12月02日 00:38, David Hill wrote:
>>>>>>>>
>>>>>>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e 
>>>>>>>> too ... compiling and I'll keep you posted.
>>>>>>>
>>>>>>> So I'm still able to reproduce this issue even with reverting 
>>>>>>> these 3 commits.  Would you have other suspect commits ? 
>>>>>>
>>>>>> Thanks for the testing. No, I don't have other suspect commits.
>>>>>>
>>>>>> Looks like somebody else it hitting your issue too (see 
>>>>>> https://www.spinics.net/lists/netdev/msg468319.html)
>>>>>>
>>>>>> But he claims the issue were fixed by using qemu 2.10.1.
>>>>>>
>>>>>> So you may:
>>>>>>
>>>>>> -try to see if qemu 2.10.1 solves your issue
>>>>> It didn't solve it for him... it's only harder to reproduce. [1]
>>>>>> -if not, try to see if commit 
>>>>>> 2ddf71e23cc246e95af72a6deed67b4a50a7b81c ("net: add notifier 
>>>>>> hooks for devmap bpf map") is the first bad commit
>>>>> I'll try to see what I can do here
>>>> I'm looking at that commit and it's been introduced before v4.13 if 
>>>> I'm not mistaken while this issue appeared between v4.13 and 
>>>> v4.14-rc1 .  Between those two releases, there're 1352 commits.
>>>> Is there a way to quickly know which commits are touching 
>>>> vhost-net, zerocopy ?
>>>>
>>>>
>>>> [ 7496.553044]  __schedule+0x2dc/0xbb0
>>>> [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>>>> [ 7496.553074]  schedule+0x3d/0x90
>>>> [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>>>> [ 7496.553100]  ? finish_wait+0x90/0x90
>>>> [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>>>> [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>>>> [ 7496.553166]  SyS_ioctl+0x79/0x90
>>>> [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>>>
>>> That vhost_net_ubuf_put_and)wait call has been changed in this 
>>> commit with the following comment:
>>>
>>> commit 0ad8b480d6ee916aa84324f69acf690142aecd0e
>>> Author: Michael S. Tsirkin <mst@redhat.com>
>>> Date:   Thu Feb 13 11:42:05 2014 +0200
>>>
>>>     vhost: fix ref cnt checking deadlock
>>>
>>>     vhost checked the counter within the refcnt before 
>>> decrementing.  It
>>>     really wanted to know that it is the one that has the last 
>>> reference, as
>>>     a way to batch freeing resources a bit more efficiently.
>>>
>>>     Note: we only let refcount go to 0 on device release.
>>>
>>>     This works well but we now access the ref counter twice so 
>>> there's a
>>>     race: all users might see a high count and decide to defer freeing
>>>     resources.
>>>     In the end no one initiates freeing resources until the last 
>>> reference
>>>     is gone (which is on VM shotdown so might happen after a 
>>> looooong time).
>>>
>>>     Let's do what we probably should have done straight away:
>>>     switch from kref to plain atomic, documenting the
>>>     semantics, return the refcount value atomically after decrement,
>>>     then use that to avoid the deadlock.
>>>
>>>     Reported-by: Qin Chuanyu <qinchuanyu@huawei.com>
>>>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>     Acked-by: Jason Wang <jasowang@redhat.com>
>>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>>>
>>>
>>>
>>> So at this point, are we hitting a deadlock when using 
>>> experimental_zcopytx ? 
>>
>> Yes. But there could be another possibility that it was not caused by 
>> vhost_net itself but other places that holds a packet.
>>
>> Thanks
>
> While bisecting, when I reach this commit 
> 46d4b68f891bee5d83a32508bfbd9778be6b1b63, the system kernel panic when 
> I run virt-customize :
>
> Message from syslogd@zappa at Dec  8 12:52:06 ...
>  kernel:[  350.016376] Kernel panic - not syncing: Fatal exception in 
> interrupt
>
> I marked that commit as bad again.   Will continue bisecting!
>

It looks like the first bad commit would be the following:

[jenkins@zappa linux-stable-new]$ sudo bash bisect.sh -g
3ece782693c4b64d588dd217868558ab9a19bfe7 is the first bad commit
commit 3ece782693c4b64d588dd217868558ab9a19bfe7
Author: Willem de Bruijn <willemb@google.com>
Date:   Thu Aug 3 16:29:38 2017 -0400

     sock: skb_copy_ubufs support for compound pages

     Refine skb_copy_ubufs to support compound pages. With upcoming TCP
     zerocopy sendmsg, such fragments may appear.

     The existing code replaces each page one for one. Splitting each
     compound page into an independent number of regular pages can result
     in exceeding limit MAX_SKB_FRAGS if data is not exactly page aligned.

     Instead, fill all destination pages but the last to PAGE_SIZE.
     Split the existing alloc + copy loop into separate stages:
     1. compute bytelength and minimum number of pages to store this.
     2. allocate
     3. copy, filling each page except the last to PAGE_SIZE bytes
     4. update skb frag array

     Signed-off-by: Willem de Bruijn <willemb@google.com>
     Signed-off-by: David S. Miller <davem@davemloft.net>

:040000 040000 f1b652be7e59b1046400cad8e6be25028a88b8e2 
6ecf86d9f06a2d98946f531f1e4cf803de071b10 M    include
:040000 040000 8420cf451fcf51f669ce81437ce7e0aacc33d2eb 
4fc8384362693e4619fab39b0a945f6f2349226b M    net

Here is the bisect log:

[root@zappa linux-stable-new]# git bisect log
git bisect start
# bad: [2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e] Linux 4.14-rc1
git bisect bad 2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e
# good: [e87c13993f16549e77abce9744af844c55154349] Linux 4.13.16
git bisect good e87c13993f16549e77abce9744af844c55154349
# good: [569dbb88e80deb68974ef6fdd6a13edb9d686261] Linux 4.13
git bisect good 569dbb88e80deb68974ef6fdd6a13edb9d686261
# good: [569dbb88e80deb68974ef6fdd6a13edb9d686261] Linux 4.13
git bisect good 569dbb88e80deb68974ef6fdd6a13edb9d686261
# bad: [aae3dbb4776e7916b6cd442d00159bea27a695c1] Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
git bisect bad aae3dbb4776e7916b6cd442d00159bea27a695c1
# good: [bf1d6b2c76eda86159519bf5c427b1fa8f51f733] Merge tag 
'staging-4.14-rc1' of 
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect good bf1d6b2c76eda86159519bf5c427b1fa8f51f733
# bad: [e833251ad813168253fef9915aaf6a8c883337b0] rxrpc: Add 
notification of end-of-Tx phase
git bisect bad e833251ad813168253fef9915aaf6a8c883337b0
# bad: [46d4b68f891bee5d83a32508bfbd9778be6b1b63] Merge tag 
'wireless-drivers-next-for-davem-2017-08-07' of 
git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next
git bisect bad 46d4b68f891bee5d83a32508bfbd9778be6b1b63
# good: [cf6c6ea352faadb15d1373d890bf857080b218a4] iwlwifi: mvm: fix the 
FIFO numbers in A000 devices
git bisect good cf6c6ea352faadb15d1373d890bf857080b218a4
# good: [65205cc465e9b37abbdbb3d595c46081b97e35bc] sctp: remove the 
typedef sctp_addiphdr_t
git bisect good 65205cc465e9b37abbdbb3d595c46081b97e35bc
# bad: [ecbd87b8430419199cc9dd91598d5552a180f558] phylink: add support 
for MII ioctl access to Clause 45 PHYs
git bisect bad ecbd87b8430419199cc9dd91598d5552a180f558
# bad: [52267790ef52d7513879238ca9fac22c1733e0e3] sock: add MSG_ZEROCOPY
git bisect bad 52267790ef52d7513879238ca9fac22c1733e0e3
# good: [04b1d4e50e82536c12da00ee04a77510c459c844] net: core: Make the 
FIB notification chain generic
git bisect good 04b1d4e50e82536c12da00ee04a77510c459c844
# good: [9217d8c2fe743f02a3ce6d430fe3b5d514fd5f1c] ipv6: Regenerate host 
route according to node pointer upon loopback up
git bisect good 9217d8c2fe743f02a3ce6d430fe3b5d514fd5f1c
# good: [0a7fd1ac2a6b316ceeb9a57a41ce0c45f6bff549] mlxsw: 
spectrum_router: Add support for route replace
git bisect good 0a7fd1ac2a6b316ceeb9a57a41ce0c45f6bff549
# good: [84b7187ca2338832e3af58eb5123c02bb6921e4e] Merge branch 
'mlxsw-Support-for-IPv6-UC-router'
git bisect good 84b7187ca2338832e3af58eb5123c02bb6921e4e
# bad: [3ece782693c4b64d588dd217868558ab9a19bfe7] sock: skb_copy_ubufs 
support for compound pages
git bisect bad 3ece782693c4b64d588dd217868558ab9a19bfe7
# good: [98ba0bd5505dcbb90322a4be07bcfe6b8a18c73f] sock: allocate skbs 
from optmem
git bisect good 98ba0bd5505dcbb90322a4be07bcfe6b8a18c73f
# first bad commit: [3ece782693c4b64d588dd217868558ab9a19bfe7] sock: 
skb_copy_ubufs support for compound pages

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Jason Wang]

On 2017年12月12日 11:53, David Hill wrote:
>
>
> On 2017-12-08 01:03 PM, David Hill wrote:
>>
>>
>> On 2017-12-07 12:13 AM, Jason Wang wrote:
>>>
>>>
>>> On 2017年12月07日 12:42, David Hill wrote:
>>>>
>>>>
>>>> On 2017-12-06 11:34 PM, David Hill wrote:
>>>>>
>>>>>
>>>>> On 2017-12-04 02:51 PM, David Hill wrote:
>>>>>>
>>>>>> On 2017-12-03 11:08 PM, Jason Wang wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2017年12月02日 00:38, David Hill wrote:
>>>>>>>>>
>>>>>>>>> Finally, I reverted 581fe0ea61584d88072527ae9fb9dcb9d1f2783e 
>>>>>>>>> too ... compiling and I'll keep you posted.
>>>>>>>>
>>>>>>>> So I'm still able to reproduce this issue even with reverting 
>>>>>>>> these 3 commits.  Would you have other suspect commits ? 
>>>>>>>
>>>>>>> Thanks for the testing. No, I don't have other suspect commits.
>>>>>>>
>>>>>>> Looks like somebody else it hitting your issue too (see 
>>>>>>> https://www.spinics.net/lists/netdev/msg468319.html)
>>>>>>>
>>>>>>> But he claims the issue were fixed by using qemu 2.10.1.
>>>>>>>
>>>>>>> So you may:
>>>>>>>
>>>>>>> -try to see if qemu 2.10.1 solves your issue
>>>>>> It didn't solve it for him... it's only harder to reproduce. [1]
>>>>>>> -if not, try to see if commit 
>>>>>>> 2ddf71e23cc246e95af72a6deed67b4a50a7b81c ("net: add notifier 
>>>>>>> hooks for devmap bpf map") is the first bad commit
>>>>>> I'll try to see what I can do here
>>>>> I'm looking at that commit and it's been introduced before v4.13 
>>>>> if I'm not mistaken while this issue appeared between v4.13 and 
>>>>> v4.14-rc1 .  Between those two releases, there're 1352 commits.
>>>>> Is there a way to quickly know which commits are touching 
>>>>> vhost-net, zerocopy ?
>>>>>
>>>>>
>>>>> [ 7496.553044]  __schedule+0x2dc/0xbb0
>>>>> [ 7496.553055]  ? trace_hardirqs_on+0xd/0x10
>>>>> [ 7496.553074]  schedule+0x3d/0x90
>>>>> [ 7496.553087]  vhost_net_ubuf_put_and_wait+0x73/0xa0 [vhost_net]
>>>>> [ 7496.553100]  ? finish_wait+0x90/0x90
>>>>> [ 7496.553115]  vhost_net_ioctl+0x542/0x910 [vhost_net]
>>>>> [ 7496.553144]  do_vfs_ioctl+0xa6/0x6c0
>>>>> [ 7496.553166]  SyS_ioctl+0x79/0x90
>>>>> [ 7496.553182]  entry_SYSCALL_64_fastpath+0x1f/0xbe
>>>>
>>>> That vhost_net_ubuf_put_and)wait call has been changed in this 
>>>> commit with the following comment:
>>>>
>>>> commit 0ad8b480d6ee916aa84324f69acf690142aecd0e
>>>> Author: Michael S. Tsirkin <mst@redhat.com>
>>>> Date:   Thu Feb 13 11:42:05 2014 +0200
>>>>
>>>>     vhost: fix ref cnt checking deadlock
>>>>
>>>>     vhost checked the counter within the refcnt before 
>>>> decrementing.  It
>>>>     really wanted to know that it is the one that has the last 
>>>> reference, as
>>>>     a way to batch freeing resources a bit more efficiently.
>>>>
>>>>     Note: we only let refcount go to 0 on device release.
>>>>
>>>>     This works well but we now access the ref counter twice so 
>>>> there's a
>>>>     race: all users might see a high count and decide to defer freeing
>>>>     resources.
>>>>     In the end no one initiates freeing resources until the last 
>>>> reference
>>>>     is gone (which is on VM shotdown so might happen after a 
>>>> looooong time).
>>>>
>>>>     Let's do what we probably should have done straight away:
>>>>     switch from kref to plain atomic, documenting the
>>>>     semantics, return the refcount value atomically after decrement,
>>>>     then use that to avoid the deadlock.
>>>>
>>>>     Reported-by: Qin Chuanyu <qinchuanyu@huawei.com>
>>>>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>>     Acked-by: Jason Wang <jasowang@redhat.com>
>>>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>>>>
>>>>
>>>>
>>>> So at this point, are we hitting a deadlock when using 
>>>> experimental_zcopytx ? 
>>>
>>> Yes. But there could be another possibility that it was not caused 
>>> by vhost_net itself but other places that holds a packet.
>>>
>>> Thanks
>>
>> While bisecting, when I reach this commit 
>> 46d4b68f891bee5d83a32508bfbd9778be6b1b63, the system kernel panic 
>> when I run virt-customize :
>>
>> Message from syslogd@zappa at Dec  8 12:52:06 ...
>>  kernel:[  350.016376] Kernel panic - not syncing: Fatal exception in 
>> interrupt
>>
>> I marked that commit as bad again.   Will continue bisecting!
>>
>
> It looks like the first bad commit would be the following:
>
> [jenkins@zappa linux-stable-new]$ sudo bash bisect.sh -g
> 3ece782693c4b64d588dd217868558ab9a19bfe7 is the first bad commit
> commit 3ece782693c4b64d588dd217868558ab9a19bfe7
> Author: Willem de Bruijn <willemb@google.com>
> Date:   Thu Aug 3 16:29:38 2017 -0400
>
>     sock: skb_copy_ubufs support for compound pages
>
>     Refine skb_copy_ubufs to support compound pages. With upcoming TCP
>     zerocopy sendmsg, such fragments may appear.
>
>     The existing code replaces each page one for one. Splitting each
>     compound page into an independent number of regular pages can result
>     in exceeding limit MAX_SKB_FRAGS if data is not exactly page aligned.
>
>     Instead, fill all destination pages but the last to PAGE_SIZE.
>     Split the existing alloc + copy loop into separate stages:
>     1. compute bytelength and minimum number of pages to store this.
>     2. allocate
>     3. copy, filling each page except the last to PAGE_SIZE bytes
>     4. update skb frag array
>
>     Signed-off-by: Willem de Bruijn <willemb@google.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
>
> :040000 040000 f1b652be7e59b1046400cad8e6be25028a88b8e2 
> 6ecf86d9f06a2d98946f531f1e4cf803de071b10 M    include
> :040000 040000 8420cf451fcf51f669ce81437ce7e0aacc33d2eb 
> 4fc8384362693e4619fab39b0a945f6f2349226b M    net
>
> Here is the bisect log:

Thanks for the hard bisecting.

Cc netdev and Willem.


>
> [root@zappa linux-stable-new]# git bisect log
> git bisect start
> # bad: [2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e] Linux 4.14-rc1
> git bisect bad 2bd6bf03f4c1c59381d62c61d03f6cc3fe71f66e
> # good: [e87c13993f16549e77abce9744af844c55154349] Linux 4.13.16
> git bisect good e87c13993f16549e77abce9744af844c55154349
> # good: [569dbb88e80deb68974ef6fdd6a13edb9d686261] Linux 4.13
> git bisect good 569dbb88e80deb68974ef6fdd6a13edb9d686261
> # good: [569dbb88e80deb68974ef6fdd6a13edb9d686261] Linux 4.13
> git bisect good 569dbb88e80deb68974ef6fdd6a13edb9d686261
> # bad: [aae3dbb4776e7916b6cd442d00159bea27a695c1] Merge 
> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
> git bisect bad aae3dbb4776e7916b6cd442d00159bea27a695c1
> # good: [bf1d6b2c76eda86159519bf5c427b1fa8f51f733] Merge tag 
> 'staging-4.14-rc1' of 
> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect good bf1d6b2c76eda86159519bf5c427b1fa8f51f733
> # bad: [e833251ad813168253fef9915aaf6a8c883337b0] rxrpc: Add 
> notification of end-of-Tx phase
> git bisect bad e833251ad813168253fef9915aaf6a8c883337b0
> # bad: [46d4b68f891bee5d83a32508bfbd9778be6b1b63] Merge tag 
> 'wireless-drivers-next-for-davem-2017-08-07' of 
> git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next
> git bisect bad 46d4b68f891bee5d83a32508bfbd9778be6b1b63
> # good: [cf6c6ea352faadb15d1373d890bf857080b218a4] iwlwifi: mvm: fix 
> the FIFO numbers in A000 devices
> git bisect good cf6c6ea352faadb15d1373d890bf857080b218a4
> # good: [65205cc465e9b37abbdbb3d595c46081b97e35bc] sctp: remove the 
> typedef sctp_addiphdr_t
> git bisect good 65205cc465e9b37abbdbb3d595c46081b97e35bc
> # bad: [ecbd87b8430419199cc9dd91598d5552a180f558] phylink: add support 
> for MII ioctl access to Clause 45 PHYs
> git bisect bad ecbd87b8430419199cc9dd91598d5552a180f558
> # bad: [52267790ef52d7513879238ca9fac22c1733e0e3] sock: add MSG_ZEROCOPY
> git bisect bad 52267790ef52d7513879238ca9fac22c1733e0e3
> # good: [04b1d4e50e82536c12da00ee04a77510c459c844] net: core: Make the 
> FIB notification chain generic
> git bisect good 04b1d4e50e82536c12da00ee04a77510c459c844
> # good: [9217d8c2fe743f02a3ce6d430fe3b5d514fd5f1c] ipv6: Regenerate 
> host route according to node pointer upon loopback up
> git bisect good 9217d8c2fe743f02a3ce6d430fe3b5d514fd5f1c
> # good: [0a7fd1ac2a6b316ceeb9a57a41ce0c45f6bff549] mlxsw: 
> spectrum_router: Add support for route replace
> git bisect good 0a7fd1ac2a6b316ceeb9a57a41ce0c45f6bff549
> # good: [84b7187ca2338832e3af58eb5123c02bb6921e4e] Merge branch 
> 'mlxsw-Support-for-IPv6-UC-router'
> git bisect good 84b7187ca2338832e3af58eb5123c02bb6921e4e
> # bad: [3ece782693c4b64d588dd217868558ab9a19bfe7] sock: skb_copy_ubufs 
> support for compound pages
> git bisect bad 3ece782693c4b64d588dd217868558ab9a19bfe7
> # good: [98ba0bd5505dcbb90322a4be07bcfe6b8a18c73f] sock: allocate skbs 
> from optmem
> git bisect good 98ba0bd5505dcbb90322a4be07bcfe6b8a18c73f
> # first bad commit: [3ece782693c4b64d588dd217868558ab9a19bfe7] sock: 
> skb_copy_ubufs support for compound pages
>
>

Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Willem de Bruijn]

>> It looks like the first bad commit would be the following:
>>
>> [jenkins@zappa linux-stable-new]$ sudo bash bisect.sh -g
>> 3ece782693c4b64d588dd217868558ab9a19bfe7 is the first bad commit
>> commit 3ece782693c4b64d588dd217868558ab9a19bfe7
>> Author: Willem de Bruijn <willemb@google.com>
>> Date:   Thu Aug 3 16:29:38 2017 -0400
>>
>>     sock: skb_copy_ubufs support for compound pages
>>
>>     Refine skb_copy_ubufs to support compound pages. With upcoming TCP
>>     zerocopy sendmsg, such fragments may appear.
>>
>>     The existing code replaces each page one for one. Splitting each
>>     compound page into an independent number of regular pages can result
>>     in exceeding limit MAX_SKB_FRAGS if data is not exactly page aligned.
>>
>>     Instead, fill all destination pages but the last to PAGE_SIZE.
>>     Split the existing alloc + copy loop into separate stages:
>>     1. compute bytelength and minimum number of pages to store this.
>>     2. allocate
>>     3. copy, filling each page except the last to PAGE_SIZE bytes
>>     4. update skb frag array
>>
>>     Signed-off-by: Willem de Bruijn <willemb@google.com>
>>     Signed-off-by: David S. Miller <davem@davemloft.net>
>>
>> :040000 040000 f1b652be7e59b1046400cad8e6be25028a88b8e2
>> 6ecf86d9f06a2d98946f531f1e4cf803de071b10 M    include
>> :040000 040000 8420cf451fcf51f669ce81437ce7e0aacc33d2eb
>> 4fc8384362693e4619fab39b0a945f6f2349226b M    net
>>
>> Here is the bisect log:
>
>
> Thanks for the hard bisecting.
>
> Cc netdev and Willem.

This is being discussed in

http://lkml.kernel.org/r/<CAF=yD-LWyCD4Y0aJ9O0e_CHLR+3JOeKicRRTEVCPxgw4XOcqGQ@mail.gmail.com>

David also previously reported this at

  https://bugzilla.kernel.org/show_bug.cgi?id=197861

which has a pointer to the above thread, too. Let's discuss this in a
single thread. I have suggested a fix there.

Thanks for bisecting. Please also test the patch in the above thread
if possible.