[Bug 75331] New: "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

[Bug 75331] New: "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

            Bug ID: 75331
           Summary: "soft lockup CPU#0 stuck for 23s" regression on 32bit
                    3.13.0+ kernels.
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.13.0+
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: virtualization_kvm@kernel-bugs.osdl.org
          Reporter: iive@yahoo.com
        Regression: No

Created attachment 134761
  --> https://bugzilla.kernel.org/attachment.cgi?id=134761&action=edit
dmesg log and kernel .config

I'm getting "soft lockup CPU#0 stuck for 23s" when running qemu 1.7.0 with kvm
on vanilla kernels 3.13.0 and above. All builds are 32 bit.

The lockup happens shortly after starting the guest OS, sometimes even before
the guest boot is complete. The cpu graph shows one core running in kernel
mode, the guest freezes, qemu is still responsive. Sometimes the lockup goes
away after a while, then comes back.
The first time I didn't kill qemu and it managed to lock my system.

I'm not using virtio/xen or other frameworks. Just bare qemu. I've tried to
remove network and sound devices but it had no measurable impact. Having -usb
and -smp 4 seems to make triggering the lockup much easier/faster but removing
them doesn't eliminate lockups. (Or lockups may be random and not related to
these options.)

With some help, I started a `git bisect` on the Linus' repository. I did manage
to locate a commit that when reversed eliminates the problem completely (tested
on 3.14.1 kernel that previously had this problem).
The commit is :
---------
commit e0f6dec35f9286e78879fe1ac92803fd69fc4fdc
Author: H. Peter Anvin <hpa@linux.intel.com>
Date:   Wed Dec 4 14:31:28 2013 -0800
    x86, bitops: Correct the assembly constraints to testing bitops
---------

I'm with gcc-4.8.2 on latest Slackware.

To this bugreport I'm attaching tgz archive with the 3.14.0 kernel `.config`
file and `dmesg` extracted from the vmcore.

On request I could provide vmcore and vmlinux of 3.14.0 compiled with debug
info. (75MB each, when gzipped/stripped by `makedumpfile`)

Qemu is compiled with: ./configure  --disable-strip
--audio-drv-list=oss,alsa,sdl --disable-bsd-user --enable-vnc
--target-list=i386-softmmu,i386-linux-user

Qemu is started with : qemu -smp 4 -usb -enable-kvm -vga cirrus -net none
-ctrl-grab -display sdl -hda guest.qcow2

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

Ivan Kalvachev <iive@yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |iive@yahoo.com
           Hardware|All                         |i386
         Regression|No                          |Yes

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

--- Comment #1 from Ivan Kalvachev <iive@yahoo.com> ---
I tried to isolate the difference in .o object files when compiling the kernel
with and without the above commit. Unfortunately there were no differences
(only in version, boot and init code).

Upon further testing the kernel with the reverted commit does seem to
experience the same lockup, it just took it more time.

The above commit is not the culprit. I'll try to redo the bisect.
Sorry for the noise.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

Roland Kletzing (devzero@web.de) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |devzero@web.de

--- Comment #2 from Roland Kletzing (devzero@web.de) ---
this may be caused by intense IO on the host if your VM is not using virtio
dataplance.

i have seen such when copying data or live migrating virtual machines

have a look at 
https://bugzilla.kernel.org/show_bug.cgi?id=199727 and
https://gitlab.com/qemu-project/qemu/-/issues/819

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 75331] "soft lockup CPU#0 stuck for 23s" regression on 32bit 3.13.0+ kernels.

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=75331

--- Comment #3 from Ivan Kalvachev (ikalvachev@gmail.com) ---
(In reply to Roland Kletzing from comment #2)
> this may be caused by intense IO on the host if your VM is not using virtio
> dataplance.

Nothing intense was running at the time.
AFAIR It could happen when both guest&host were mostly idle.

The issue was also completely gone when using 64bit kernel/Slackware and the
same 32bit guest. I don't remember if I've tried the same qemu binary.

Unfortunately, I no longer have the kernel memory image dump.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.