regression: massive trouble with fpu rework

regression: massive trouble with fpu rework

[Mike Galbraith]

Hi Ingo,

My i7-4790 box is having one hell of a time with this merge window, is
dead in the water.  The netconsole log below is v4.1-7254-gc13c81006314,
but trouble begins at bisected point much earlier.  If I turn off kvm,
such that I can kinda sorta boot, systemd says many services "enter
failed state", box is pretty much a doorstop.  Though I can get to a
prompt, I can't login.  If kvm is enabled, it explodes as soon as it
autoloads (wtf does it do that when it's not being used?)

Bisecting to the beginning of my woes takes me to the below.  Before
that, it doesn't matter if kvm is enabled or not, all is well.  Below
the current gripage with kvm disabled, find the kvm explosion, and
another explosion as I approach the beginning of my box's woes.

067051ccd209623cb56152cf4cb06616ee2bcc5c is the first bad commit
commit 067051ccd209623cb56152cf4cb06616ee2bcc5c
Author: Ingo Molnar <mingo@kernel.org>
Date:   Sat Apr 25 08:27:44 2015 +0200

    x86/fpu: Do system-wide setup from fpu__detect()
    
    fpu__cpu_init() is called on every CPU, so it is the wrong place
    to call fpu__init_system() from. Call it from fpu__detect():
    this is early CPU init code, but we already have CPU features detected,
    so we can call the system-wide FPU init code from here.
    
    Reviewed-by: Borislav Petkov <bp@alien8.de>
    Cc: Andy Lutomirski <luto@amacapital.net>
    Cc: Dave Hansen <dave.hansen@linux.intel.com>
    Cc: Fenghua Yu <fenghua.yu@intel.com>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Oleg Nesterov <oleg@redhat.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Signed-off-by: Ingo Molnar <mingo@kernel.org>

:040000 040000 b85f9a16eea19a4ec974c00ea53144a3d11d4e99 697905b73e3f8931a28588c728da3ab1a7289afa M      arch

git bisect start
# good: [cfe3eceb7a2eb91284d5605c5315249bb165e9d3] Merge branch 'x86-efi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good cfe3eceb7a2eb91284d5605c5315249bb165e9d3
# bad: [650ec5a6bd5df4ab0c9ef38d05b94cd82fb99ad8] Merge branch 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 650ec5a6bd5df4ab0c9ef38d05b94cd82fb99ad8
# bad: [c4d6ee6e2e52ec604cc1d76877791f8e8f5c79b5] x86/fpu: Remove failure paths from fpstate-alloc low level functions
git bisect bad c4d6ee6e2e52ec604cc1d76877791f8e8f5c79b5
# good: [db2b1d3ad1cdae9f268d6db54b6127b09933da3d] x86/fpu: Use 'struct fpu' in fpstate_alloc_init()
git bisect good db2b1d3ad1cdae9f268d6db54b6127b09933da3d
# good: [e9dbfd673a9cc9d3ef90e3ea4e136a833dab3674] x86/fpu: Move legacy check to fpu__init_system_xstate()
git bisect good e9dbfd673a9cc9d3ef90e3ea4e136a833dab3674
# bad: [7218e8b723dc9ceb71cf2fbd3d019e9e11b7d3cf] x86/fpu: Factor out fpu__init_system_generic()
git bisect bad 7218e8b723dc9ceb71cf2fbd3d019e9e11b7d3cf
# good: [064e51e3c8aee6cfb03ab75e9f9551db3924eb07] x86/fpu: Clean up eager_fpu_init() and rename it to fpu__ctx_switch_init()
git bisect good 064e51e3c8aee6cfb03ab75e9f9551db3924eb07
# good: [3960fccf2e22c3b3d61bd5a45b6172e66e660d8b] x86/fpu: Call fpu__init_cpu_ctx_switch() from fpu__init_cpu()
git bisect good 3960fccf2e22c3b3d61bd5a45b6172e66e660d8b
# bad: [7202ab46f7392265c1ecbf03f600393bf32a8bdf] x86/fpu: Remove fpu__init_cpu_ctx_switch() call from fpu__init_system()
git bisect bad 7202ab46f7392265c1ecbf03f600393bf32a8bdf
# bad: [067051ccd209623cb56152cf4cb06616ee2bcc5c] x86/fpu: Do system-wide setup from fpu__detect()
git bisect bad 067051ccd209623cb56152cf4cb06616ee2bcc5c
# first bad commit: [067051ccd209623cb56152cf4cb06616ee2bcc5c] x86/fpu: Do system-wide setup from fpu__detect()


[    2.411280] ------------[ cut here ]------------
[    2.412198] WARNING: CPU: 3 PID: 1 at ./arch/x86/include/asm/fpu/internal.h:366 copy_kernel_to_xregs.part.8+0x1e/0x27()
[    2.413132] Modules linked in:
[    2.414003] CPU: 3 PID: 1 Comm: init Tainted: G        W       4.1.0-master #279
[    2.414872] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.415745]  ffffffff817f8db8 ffff88040c91fea8 ffffffff8159726c 0000000000000001
[    2.416633]  0000000000000000 ffff88040c91fee8 ffffffff81052bca 00000000ffffffff
[    2.417524]  ffff88040ca18600 ffff88040ca18600 0000000000000000 00007f77d5254148
[    2.418411] Call Trace:
[    2.419286]  [<ffffffff8159726c>] dump_stack+0x45/0x57
[    2.420159]  [<ffffffff81052bca>] warn_slowpath_common+0x8a/0xc0
[    2.421017]  [<ffffffff81052cba>] warn_slowpath_null+0x1a/0x20
[    2.421860]  [<ffffffff81591bf5>] copy_kernel_to_xregs.part.8+0x1e/0x27
[    2.422702]  [<ffffffff8100dfe9>] fpu__restore+0xf9/0x100
[    2.423536]  [<ffffffff810045c3>] do_device_not_available+0x33/0x40
[    2.424356]  [<ffffffff8159f178>] device_not_available+0x18/0x20
[    2.425159] ---[ end trace cb88537fdc8fa202 ]---
[    2.426492] ------------[ cut here ]------------
[    2.427289] WARNING: CPU: 3 PID: 1 at ./arch/x86/include/asm/fpu/internal.h:337 __switch_to+0x586/0x5d0()
[    2.428100] Modules linked in:
[    2.428907] CPU: 3 PID: 1 Comm: systemd Tainted: G        W       4.1.0-master #279
[    2.429943] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.431370]  ffffffff817f8db8 ffff880409a0fd10 ffffffff8159726c 0000000000000001
[    2.432801]  0000000000000000 ffff880409a0fd50 ffffffff81052bca ffff88040ca185c0
[    2.434233]  ffff880409a01c80 ffff88040ca18000 ffff88041ecd2080 0000000000000003
[    2.435662] Call Trace:
[    2.437067]  <UNK> 
[    2.437077] ---[ end trace cb88537fdc8fa203 ]---
[    2.440704] random: systemd urandom read with 49 bits of entropy available
[    2.441642] systemd[1]: systemd 210 running in system mode. (+PAM -LIBWRAP +AUDIT +SELINUX -IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ +SECCOMP +APPARMOR)
[    2.442612] systemd[1]: Detected architecture 'x86-64'.
[    2.443411] systemd[1]: Running in initial RAM disk.
[    2.447158] systemd[1]: Inserted module 'autofs4'
[    2.448014] systemd[1]: Set hostname to <homer>.
[    2.449122] ------------[ cut here ]------------
[    2.449964] WARNING: CPU: 2 PID: 1 at ./arch/x86/include/asm/fpu/internal.h:337 fpu__copy+0xd3/0x120()
[    2.450785] Modules linked in: autofs4
[    2.451591] CPU: 2 PID: 1 Comm: systemd Tainted: G        W       4.1.0-master #279
[    2.452404] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.453215]  ffffffff817f8db8 ffff88040c91fd58 ffffffff8159726c 0000000000000001
[    2.454025]  0000000000000000 ffff88040c91fd98 ffffffff81052bca 00000000ffffffff
[    2.454810]  ffff8800dca5a280 ffff88040ca18000 0000000000000000 ffff8800dca59c80
[    2.455590] Call Trace:
[    2.456363]  [<ffffffff8159726c>] dump_stack+0x45/0x57
[    2.457121]  [<ffffffff81052bca>] warn_slowpath_common+0x8a/0xc0
[    2.457867]  [<ffffffff81052cba>] warn_slowpath_null+0x1a/0x20
[    2.458595]  [<ffffffff8100e193>] fpu__copy+0xd3/0x120
[    2.459312]  [<ffffffff8100d48e>] arch_dup_task_struct+0x2e/0x40
[    2.460015]  [<ffffffff810509a0>] copy_process.part.42+0x120/0x1710
[    2.460721]  [<ffffffff8116df05>] ? __sb_end_write+0x35/0x70
[    2.461418]  [<ffffffff8117d69a>] ? iterate_dir+0x5a/0x110
[    2.462114]  [<ffffffff81052145>] do_fork+0xd5/0x330
[    2.462798]  [<ffffffff8117db4d>] ? SyS_getdents+0x9d/0xe0
[    2.463475]  [<ffffffff8117d820>] ? fillonedir+0xd0/0xd0
[    2.464144]  [<ffffffff81052426>] SyS_clone+0x16/0x20
[    2.464815]  [<ffffffff8159d9db>] entry_SYSCALL_64_fastpath+0x16/0x6e
[    2.465479] ---[ end trace cb88537fdc8fa204 ]---
[    2.469752] ------------[ cut here ]------------
[    2.470483] WARNING: CPU: 1 PID: 98 at ./arch/x86/include/asm/fpu/internal.h:366 copy_kernel_to_xregs.part.0+0x1e/0x27()
[    2.471173] Modules linked in: autofs4
[    2.471834] CPU: 1 PID: 98 Comm: dracut-rootfs-g Tainted: G        W       4.1.0-master #279
[    2.472502] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.473169]  ffffffff817f8db8 ffff880409be3da0 ffffffff8159726c 0000000000000001
[    2.473844]  0000000000000000 ffff880409be3de0 ffffffff81052bca 0000000000000003
[    2.474515]  00007ffdf76bc580 ffff880409b75b80 ffff880409b75b80 ffff880409b75580
[    2.475182] Call Trace:
[    2.475841]  [<ffffffff8159726c>] dump_stack+0x45/0x57
[    2.476505]  [<ffffffff81052bca>] warn_slowpath_common+0x8a/0xc0
[    2.477177]  [<ffffffff81052cba>] warn_slowpath_null+0x1a/0x20
[    2.477838]  [<ffffffff81591c43>] copy_kernel_to_xregs.part.0+0x1e/0x27
[    2.478500]  [<ffffffff8100ef11>] __fpu__restore_sig+0x251/0x3f0
[    2.479162]  [<ffffffff810551ca>] ? do_wait+0x11a/0x220
[    2.479822]  [<ffffffff8105cbef>] ? recalc_sigpending+0x1f/0x60
[    2.480485]  [<ffffffff8100f2cd>] fpu__restore_sig+0x4d/0x60
[    2.481148]  [<ffffffff81003282>] restore_sigcontext+0x132/0x150
[    2.481796]  [<ffffffff81003f6d>] sys_rt_sigreturn+0x9d/0xc0
[    2.482437]  [<ffffffff8159dd33>] stub_rt_sigreturn+0x23/0x23
[    2.483071]  [<ffffffff8159d9db>] ? entry_SYSCALL_64_fastpath+0x16/0x6e
[    2.483702] ---[ end trace cb88537fdc8fa205 ]---
[    2.501200] ------------[ cut here ]------------
[    2.501925] WARNING: CPU: 2 PID: 0 at ./arch/x86/include/asm/fpu/internal.h:366 __switch_to+0x3dc/0x5d0()
[    2.502603] Modules linked in: autofs4
[    2.503242] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W       4.1.0-master #279
[    2.503905] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.504576]  ffffffff817f8db8 ffff88040c91fd00 ffffffff8159726c 0000000000000001
[    2.505253]  0000000000000000 ffff88040c91fd40 ffffffff81052bca 00000000c0000100
[    2.505901]  ffff88040ca18000 ffff88040ca5b900 ffff88041ec92080 ffff88040ca18600
[    2.506530] Call Trace:
[    2.507149]  <UNK> 
[    2.507155] ---[ end trace cb88537fdc8fa206 ]---
[    2.510183] systemd[1]: Expecting device dev-disk-by\x2duuid-63d8c9f4\x2dc094\x2d4dd8\x2d96cd\x2d23c5e91d0322.device...
[    2.511523] systemd[1]: Starting Timers.
[    2.512962] systemd[1]: Reached target Timers.
[    2.513608] systemd[1]: Starting -.slice.
[    2.514900] systemd[1]: Created slice -.slice.
[    2.515520] systemd[1]: Starting udev Control Socket.
[    2.516849] systemd[1]: Listening on udev Control Socket.
[    2.517473] systemd[1]: Starting udev Kernel Socket.
[    2.518737] systemd[1]: Listening on udev Kernel Socket.
[    2.519351] systemd[1]: Starting Journal Socket.
[    2.520683] systemd[1]: Listening on Journal Socket.
[    2.521321] systemd[1]: Starting Sockets.
[    2.522595] systemd[1]: Reached target Sockets.
[    2.523218] systemd[1]: Starting System Slice.
[    2.524566] systemd[1]: Created slice System Slice.
[    2.525247] systemd[1]: Starting dracut cmdline hook...
[    2.526651] systemd[1]: Starting system-systemd\x2dfsck.slice.
[    2.528034] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    2.528792] systemd[1]: Starting Load Kernel Modules...
[    2.530325] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[    2.531235] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    2.531278] sd 1:0:0:0: Attached scsi generic sg1 type 0
[    2.531297] sd 4:0:0:0: Attached scsi generic sg2 type 0
[    2.531316] sd 5:0:0:0: Attached scsi generic sg3 type 0
[    2.534490] systemd[1]: Starting Journal Service...
[    2.536022] systemd[1]: Starting Slices.
[    2.537140] systemd-journald[154]: Failed to set file attributes: Inappropriate ioctl for device
[    2.538285] systemd[1]: Reached target Slices.
[    2.538946] systemd[1]: Starting Setup Virtual Console...
[    2.540465] systemd[1]: Starting Swap.
[    2.542612] systemd[1]: Reached target Swap.
[    2.543709] systemd[1]: Starting Local File Systems.
[    2.545112] systemd[1]: Reached target Local File Systems.
[    2.546961] systemd[1]: Started Load Kernel Modules.
[    2.548995] systemd[1]: Started Create list of required static device nodes for the current kernel.
[    2.550572] systemd[1]: Started Setup Virtual Console.
[    2.552908] systemd[1]: Starting Create static device nodes in /dev...
[    2.554608] systemd[1]: Starting Apply Kernel Variables...
[    2.556948] systemd[1]: Started Create static device nodes in /dev.
[    2.558383] systemd[1]: Started Apply Kernel Variables.
[    2.559749] systemd[1]: Started dracut cmdline hook.
[    2.561210] systemd[1]: Started Journal Service.
[    2.585519] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input7
[    2.586779] ACPI: Power Button [PWRB]
[    2.587983] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input8
[    2.588018] ACPI: Power Button [PWRF]
[    2.592267] [drm] Initialized drm 1.1.0 20060810
[    2.594196] wmi: Mapper loaded
[    2.640007] clocksource: Switched to clocksource tsc
[    2.868141] random: nonblocking pool is initialized
[    2.937645] EXT4-fs (sdc3): mounted filesystem with ordered data mode. Opts: data=ordered
[    3.082729] systemd-journald[154]: Received SIGTERM from PID 1 (systemd).
[    4.944246] device-mapper: uevent: version 1.0.3
[    4.944309] device-mapper: ioctl: 4.31.0-ioctl (2015-3-12) initialised: dm-devel@redhat.com
[    5.540586] systemd-readahead[399]: Bumped block_nr parameter of 8:32 to 20480. This is a temporary hack and should be removed one day.
[    6.851888] ACPI Warning: SystemIO range 0x000000000000F000-0x000000000000F01F conflicts with OpRegion 0x000000000000F000-0x000000000000F00F (\_SB_.PCI0.SBUS.SMBI) (20150515/utaddress-254)
[    6.852656] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    6.931752] thermal LNXTHERM:00: registered as thermal_zone0
[    6.932515] ACPI: Thermal Zone [TZ00] (28 C)
[    6.933477] thermal LNXTHERM:01: registered as thermal_zone1
[    6.934206] ACPI: Thermal Zone [TZ01] (30 C)
[    7.151702] iTCO_vendor_support: vendor-support=0
[    7.152822] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[    7.153543] iTCO_wdt: Found a Lynx Point TCO device (Version=2, TCOBASE=0x1860)
[    7.154304] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[    7.561298] EXT4-fs (sdc3): re-mounted. Opts: acl,user_xattr
[    7.828308] EXT4-fs (sdd1): mounted filesystem with writeback data mode. Opts: data=writeback,acl,user_xattr
[    8.105380] AVX2 version of gcm_enc/dec engaged.
[    8.106197] AES CTR mode by8 optimization enabled
[    8.371083] Adding 2104316k swap on /dev/sdc2.  Priority:-1 extents:1 across:2104316k FS
[    8.372290] snd_hda_intel 0000:00:1b.0: enabling device (0000 -> 0002)
[    8.372369] snd_hda_intel 0000:01:00.1: enabling device (0000 -> 0002)
[    8.372392] snd_hda_intel 0000:01:00.1: Disabling MSI
[    8.372395] snd_hda_intel 0000:01:00.1: Handle VGA-switcheroo audio client
[    8.504246] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC887-VD: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line
[    8.504247] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    8.504248] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    8.504248] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    8.504249] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    8.504250] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    8.504250] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    8.504251] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[    8.515534] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[    8.515563] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[    8.515587] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input11
[    8.515612] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[    8.515636] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[    9.053824] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input14
[    9.055403] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input15
[    9.056681] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input16
[    9.057436] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input17
[    9.115047] FAT-fs (sdc1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   10.043873] systemd-journald[403]: Received request to flush runtime journal from PID 1
[   10.340931] systemd[1]: Unit apparmor.service entered failed state.
[   11.072589] systemd[1]: set_kthread_prio.service: control process exited, code=killed status=11
[   11.096816] systemd[1]: Unit set_kthread_prio.service entered failed state.
[   11.133099] systemd[1]: Unit kdump.service entered failed state.
[   11.180579] systemd[1]: Unit SuSEfirewall2_init.service entered failed state.
[   11.344489] systemd[1]: display-manager.service: control process exited, code=killed status=11
[   11.368512] systemd[1]: Unit display-manager.service entered failed state.
[   12.062671] NET: Registered protocol family 17
[   12.517609] systemd[1]: Unit wickedd-dhcp4.service entered failed state.
[   12.760012] systemd[1]: wickedd-dhcp4.service holdoff time over, scheduling restart.
[   13.008371] systemd[1]: Unit wickedd.service entered failed state.
[   13.259794] systemd[1]: wickedd.service holdoff time over, scheduling restart.
[   14.324202] systemd[1]: Unit wickedd-nanny.service entered failed state.
[   14.396138] systemd[1]: Unit wickedd-auto4.service entered failed state.
[   14.504142] systemd[1]: Unit wickedd-dhcp6.service entered failed state.
[   25.551726] sysrq: SysRq : Power Off
[   25.567346] sd 5:0:0:0: [sdd] Synchronizing SCSI cache
[   25.568069] sd 5:0:0:0: [sdd] Stopping disk
[   25.979200] sd 4:0:0:0: [sdc] Synchronizing SCSI cache
[   26.044734] sd 4:0:0:0: [sdc] Stopping disk
[   26.523508] sd 1:0:0:0: [sdb] Synchronizing SCSI cache
[   26.524253] sd 1:0:0:0: [sdb] Stopping disk
[   26.928454] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[   26.933453] sd 0:0:0:0: [sda] Stopping disk

[    9.422011] invalid opcode: 0000 [#1] SMP 
[    9.422806] Modules linked in: kvm_intel(+) kvm snd_hda_codec_hdmi(+) crct10dif_pclmul snd_hda_codec_realtek snd_hda_codec_generic crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_hda_core snd_pcm aesni_intel aes_x86_64 lrw gf128mul snd_timer iTCO_wdt tpm_infineon snd glue_helper mei_me iTCO_vendor_support mei i2c_i801 tpm_tis lpc_ich joydev ablk_helper thermal battery processor mfd_core fan tpm soundcore cryptd intel_smartconnect serio_raw dm_mod efivarfs nouveau mxm_wmi wmi i2c_algo_bit drm_kms_helper ttm drm video thermal_sys button sg autofs4
[    9.425350] CPU: 1 PID: 480 Comm: systemd-udevd Tainted: G        W       4.1.0-master #268
[    9.426174] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    9.427000] task: ffff88040a658000 ti: ffff88040a660000 task.ti: ffff88040a660000
[    9.427828] RIP: 0010:[<ffffffffa041c811>]  [<ffffffffa041c811>] kvm_arch_init+0xf1/0x1c0 [kvm]
[    9.428679] RSP: 0018:ffff88040a663c68  EFLAGS: 00010246
[    9.429517] RAX: 0000000077fafbff RBX: 0000000000000000 RCX: 0000000000000000
[    9.430358] RDX: 0000000080000001 RSI: ffffffffa0455c60 RDI: ffffffffa0455c40
[    9.431198] RBP: ffff88040a663c88 R08: 0000000000000080 R09: 0000000000000000
[    9.432037] R10: 00000000000006c8 R11: 000000000001cf69 R12: ffffffffa0920160
[    9.432879] R13: 0000000000000000 R14: 0000000000004480 R15: ffffffffa09207e0
[    9.433715] FS:  00007f24f12c7880(0000) GS:ffff88041ec40000(0000) knlGS:0000000000000000
[    9.434557] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    9.435397] CR2: 00007f24f12d2000 CR3: 000000040a64f000 CR4: 00000000001007e0
[    9.436242] Stack:
[    9.437059]  ffffffffa09207e0 ffffffff81a12040 ffff8800dbc06e00 0000000000000040
[    9.437876]  ffff88040a663cd8 ffffffffa0407004 ffff88040a663d18 ffffffff00000018
[    9.438684]  ffff88040a663cf8 ffffffff81a12040 ffff8800dbc06e00 0000000000000000
[    9.439489] Call Trace:
[    9.440281]  [<ffffffffa0407004>] kvm_init+0x24/0x2b0 [kvm]
[    9.441074]  [<ffffffffa0341c5a>] ? vmx_check_processor_compat+0x60/0x60 [kvm_intel]
[    9.441874]  [<ffffffffa0341c7b>] vmx_init+0x21/0x3a6 [kvm_intel]
[    9.442661]  [<ffffffff810002fc>] do_one_initcall+0xbc/0x1f0
[    9.443436]  [<ffffffff8116ecf5>] ? kmem_cache_alloc_trace+0x35/0x130
[    9.444210]  [<ffffffff815c2a74>] do_init_module+0x61/0x1cc
[    9.444979]  [<ffffffff810c6693>] load_module+0x1cc3/0x2590
[    9.445744]  [<ffffffff810c2590>] ? store_uevent+0x40/0x40
[    9.446503]  [<ffffffff810c7136>] SyS_finit_module+0x86/0xb0
[    9.447262]  [<ffffffff815cd29b>] system_call_fastpath+0x16/0x6e
[    9.448018] Code: 25 25 2e 04 00 e8 00 38 00 00 e8 cb 4f ff ff 48 c7 c7 40 5c 45 a0 e8 af 81 cf e0 48 8b 05 dc bd 6b e1 a9 00 00 00 04 74 15 31 c9 <0f> 01 d0 48 c1 e2 20 89 c0 48 01 d0 48 89 05 44 00 04 00 e8 27 
[    9.449791] RIP  [<ffffffffa041c811>] kvm_arch_init+0xf1/0x1c0 [kvm]
[    9.450595]  RSP <ffff88040a663c68>
[    9.451403] ---[ end trace cb88537fdc8fa207 ]---
[    9.452740] Kernel panic - not syncing: Fatal exception
[    9.453576] Kernel Offset: disabled
[    9.454381] Rebooting in 60 seconds..


[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/fpu/xsave.c:646 fpu__init_system_xstate+0x1da/0x440()
[    0.000000] x86/fpu: XSTATE_CPUID missing!
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.1.0-master #270
[    0.000000]  ffffffff817f0b3b ffffffff81a03dc8 ffffffff815c5088 000000000000002e
[    0.000000]  ffffffff81a03e18 ffffffff81a03e08 ffffffff8105432a 0000000000000000
[    0.000000]  0000000000000008 ffffffff81603f80 0000000001000000 00000000cbf3b260
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815c5088>] dump_stack+0x45/0x57
[    0.000000]  [<ffffffff8105432a>] warn_slowpath_common+0x8a/0xc0
[    0.000000]  [<ffffffff810543a6>] warn_slowpath_fmt+0x46/0x50
[    0.000000]  [<ffffffff8100f7aa>] fpu__init_system_xstate+0x1da/0x440
[    0.000000]  [<ffffffff8100d680>] fpu__init_system+0x90/0x1c0
[    0.000000]  [<ffffffff81afffed>] early_cpu_init+0xe6/0x144
[    0.000000]  [<ffffffff81afc293>] setup_arch+0x95/0xc4b
[    0.000000]  [<ffffffff8109b8af>] ? vprintk_default+0x1f/0x30
[    0.000000]  [<ffffffff81af9c4b>] start_kernel+0x90/0x41f
[    0.000000]  [<ffffffff81b49b2f>] ? memblock_reserve+0x4c/0x51
[    0.000000]  [<ffffffff81af95ad>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81af969a>] x86_64_start_kernel+0xeb/0xef
[    0.000000] ---[ end trace cb88537fdc8fa200 ]---
[    0.000000] x86/fpu: Using 'lazy' FPU context switches.

....
[    2.454087] console [netcon0] enabled
[    2.454902] netconsole: network logging started
[    2.455758] rtc_cmos 00:02: setting system clock to 2015-06-27 04:58:45 UTC (1435381125)
[    2.456816] Freeing unused kernel memory: 1164K (ffffffff81ae2000 - ffffffff81c05000)
[    2.457625] Write protecting the kernel read-only data: 10240k
[    2.458682] Freeing unused kernel memory: 180K (ffff8800015d3000 - ffff880001600000)
[    2.459728] Freeing unused kernel memory: 1208K (ffff8800018d2000 - ffff880001a00000)
[    2.460638] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.460638] 
[    2.462272] CPU: 6 PID: 1 Comm: init Tainted: G        W       4.1.0-master #277
[    2.463086] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.463906]  ffff880409a78000 ffff88040ca1bc88 ffffffff815c50d2 00000000000000ea
[    2.464735]  ffffffff817fe7e8 ffff88040ca1bd08 ffffffff815c0a7e ffff88040ca1bce8
[    2.465553]  ffff880400000010 ffff88040ca1bd18 ffff88040ca1bcb8 ffff88040ca10000
[    2.466361] Call Trace:
[    2.467152]  [<ffffffff815c50d2>] dump_stack+0x45/0x57
[    2.467935]  [<ffffffff815c0a7e>] panic+0xbb/0x1df
[    2.468701]  [<ffffffff81055f86>] do_exit+0xac6/0xad0
[    2.469462]  [<ffffffff81056c3f>] do_group_exit+0x3f/0xa0
[    2.470215]  [<ffffffff810615f7>] get_signal+0x1d7/0x5d0
[    2.470953]  [<ffffffff8105f435>] ? complete_signal+0x105/0x210
[    2.471679]  [<ffffffff81002da0>] do_signal+0x20/0x120
[    2.472396]  [<ffffffff8105f998>] ? __send_signal.constprop.25+0x138/0x3e0
[    2.473115]  [<ffffffff81060434>] ? force_sig_info+0xd4/0xf0
[    2.473830]  [<ffffffff81002f08>] do_notify_resume+0x68/0x80
[    2.474544]  [<ffffffff815cc5f3>] retint_signal+0x3d/0x7a
[    2.475285] Kernel Offset: disabled
[    2.476011] Rebooting in 60 seconds..

[all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Sat, 2015-06-27 at 08:25 +0200, Mike Galbraith wrote:
> Hi Ingo,
> 
> My i7-4790 box is having one hell of a time with this merge window, is
> dead in the water.

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

	-Mike

Re: regression: massive trouble with fpu rework

[Ingo Molnar]

* Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> Hi Ingo,
> 
> My i7-4790 box is having one hell of a time with this merge window, is
> dead in the water.  The netconsole log below is v4.1-7254-gc13c81006314,
> but trouble begins at bisected point much earlier.  If I turn off kvm,
> such that I can kinda sorta boot, systemd says many services "enter
> failed state", box is pretty much a doorstop.  Though I can get to a
> prompt, I can't login.  If kvm is enabled, it explodes as soon as it
> autoloads (wtf does it do that when it's not being used?)
> 
> Bisecting to the beginning of my woes takes me to the below.  Before
> that, it doesn't matter if kvm is enabled or not, all is well.  Below
> the current gripage with kvm disabled, find the kvm explosion, and
> another explosion as I approach the beginning of my box's woes.
> 
> 067051ccd209623cb56152cf4cb06616ee2bcc5c is the first bad commit
> commit 067051ccd209623cb56152cf4cb06616ee2bcc5c
> Author: Ingo Molnar <mingo@kernel.org>
> Date:   Sat Apr 25 08:27:44 2015 +0200
> 
>     x86/fpu: Do system-wide setup from fpu__detect()
>     
>     fpu__cpu_init() is called on every CPU, so it is the wrong place
>     to call fpu__init_system() from. Call it from fpu__detect():
>     this is early CPU init code, but we already have CPU features detected,
>     so we can call the system-wide FPU init code from here.
>     
>     Reviewed-by: Borislav Petkov <bp@alien8.de>
>     Cc: Andy Lutomirski <luto@amacapital.net>
>     Cc: Dave Hansen <dave.hansen@linux.intel.com>
>     Cc: Fenghua Yu <fenghua.yu@intel.com>
>     Cc: H. Peter Anvin <hpa@zytor.com>
>     Cc: Linus Torvalds <torvalds@linux-foundation.org>
>     Cc: Oleg Nesterov <oleg@redhat.com>
>     Cc: Peter Zijlstra <peterz@infradead.org>
>     Cc: Thomas Gleixner <tglx@linutronix.de>
>     Signed-off-by: Ingo Molnar <mingo@kernel.org>

Just as a quick workaround, if you add back a per CPU init fpu__init_system() 
call, as per the disgusting hack below, do things get happier?

( You might trigger a few WARN_ON_ONCE() whinges, especially if you have 
  CONFIG_X86_DEBUG_FPU=y, but those should be one-time warnings that are not 
  fatal. )

Totally untested, unfortunately.

My theory of the bug is that there is something that needs to be set up per CPU, 
which is a side effect of fpu__init_system(), and which the new fpu__init_cpu() 
does not capture. If this patch helps then the real fix would be to figure out 
that side effect.

Thanks,

	Ingo

 arch/x86/kernel/fpu/init.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c
index fc878fee6a51..421babb08fe6 100644
--- a/arch/x86/kernel/fpu/init.c
+++ b/arch/x86/kernel/fpu/init.c
@@ -4,6 +4,9 @@
 #include <asm/fpu/internal.h>
 #include <asm/tlbflush.h>
 
+#undef __init
+#define __init
+
 /*
  * Initialize the TS bit in CR0 according to the style of context-switches
  * we are using:
@@ -44,13 +47,18 @@ static void fpu__init_cpu_generic(void)
 /*
  * Enable all supported FPU features. Called when a CPU is brought online:
  */
-void fpu__init_cpu(void)
+static void __fpu__init_cpu(void)
 {
 	fpu__init_cpu_generic();
 	fpu__init_cpu_xstate();
 	fpu__init_cpu_ctx_switch();
 }
 
+void fpu__init_cpu(void)
+{
+	fpu__init_system(NULL);
+}
+
 /*
  * The earliest FPU detection code.
  *
@@ -267,13 +275,14 @@ static void __init fpu__init_system_ctx_switch(void)
  */
 void __init fpu__init_system(struct cpuinfo_x86 *c)
 {
-	fpu__init_system_early_generic(c);
+	if (c)
+		fpu__init_system_early_generic(c);
 
 	/*
 	 * The FPU has to be operational for some of the
 	 * later FPU init activities:
 	 */
-	fpu__init_cpu();
+	__fpu__init_cpu();
 
 	/*
 	 * But don't leave CR0::TS set yet, as some of the FPU setup

Re: [all better] Re: regression: massive trouble with fpu rework

[Ingo Molnar]

* Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Sat, 2015-06-27 at 08:25 +0200, Mike Galbraith wrote:
> > Hi Ingo,
> > 
> > My i7-4790 box is having one hell of a time with this merge window, is
> > dead in the water.
> 
> BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

Ok, that's interesting. Mind explaining it a bit more verbosely - which setting is 
causing what?

This would suggest sensitivity on CPUID details, i.e. that doing 
fpu__init_system() before other CPU init sequences is causing the bug.

Does the patch below perhaps make a difference? (I'd suggest to apply it _without_ 
the other patch I sent.)

Thanks,

	Ingo

 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 9fc5e3d9d9c8..922c5e0cea4c 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 	cpu_detect(c);
 	get_cpu_vendor(c);
 	get_cpu_cap(c);
-	fpu__init_system(c);
 
 	if (this_cpu->c_early_init)
 		this_cpu->c_early_init(c);
@@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		this_cpu->c_bsp_init(c);
 
 	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
+	fpu__init_system(c);
 }
 
 void __init early_cpu_init(void)

Re: [all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Sat, 2015-06-27 at 10:25 +0200, Ingo Molnar wrote:
> * Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 
> > On Sat, 2015-06-27 at 08:25 +0200, Mike Galbraith wrote:
> > > Hi Ingo,
> > > 
> > > My i7-4790 box is having one hell of a time with this merge window, is
> > > dead in the water.
> > 
> > BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.
> 
> Ok, that's interesting. Mind explaining it a bit more verbosely - which setting is 
> causing what?

That BIOS setting is annotated with the helpful text "Disabled for
Windows XP".  It makes box say interesting things during boot, like...

        x86/fpu: XSTATE_CPUID missing!


..or with HEAD, it triggers warning..

        if (boot_cpu_data.cpuid_level < XSTATE_CPUID) {
                WARN_ON_FPU(1);
                return;
        }

..and all kinds of bad juju follows.  I have no idea what the thing does
beyond what I can interpolate from the word 'limit'.

> This would suggest sensitivity on CPUID details, i.e. that doing 
> fpu__init_system() before other CPU init sequences is causing the bug.
> 
> Does the patch below perhaps make a difference? (I'd suggest to apply it _without_ 
> the other patch I sent.)

Yup, that made it not care about the BIOS setting.. again.

> Thanks,
> 
> 	Ingo
> 
>  arch/x86/kernel/cpu/common.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 9fc5e3d9d9c8..922c5e0cea4c 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
>  	cpu_detect(c);
>  	get_cpu_vendor(c);
>  	get_cpu_cap(c);
> -	fpu__init_system(c);
>  
>  	if (this_cpu->c_early_init)
>  		this_cpu->c_early_init(c);
> @@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
>  		this_cpu->c_bsp_init(c);
>  
>  	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
> +	fpu__init_system(c);
>  }
>  
>  void __init early_cpu_init(void)

Re: [all better] Re: regression: massive trouble with fpu rework

[Borislav Petkov]

On Sat, Jun 27, 2015 at 10:55:28AM +0200, Mike Galbraith wrote:
> Yup, that made it not care about the BIOS setting.. again.

Does it say

	"x86/fpu: Legacy x87 FPU detected."

with Ingo's patch?

Or do you see that "x86/fpu: Enabled xstate features... " print out from
the end of fpu__init_system_xstate()?

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: [all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Sat, 2015-06-27 at 11:37 +0200, Borislav Petkov wrote:
> On Sat, Jun 27, 2015 at 10:55:28AM +0200, Mike Galbraith wrote:
> > Yup, that made it not care about the BIOS setting.. again.
> 
> Does it say
> 
> 	"x86/fpu: Legacy x87 FPU detected."
> 
> with Ingo's patch?

Nope.

> Or do you see that "x86/fpu: Enabled xstate features... " print out from
> the end of fpu__init_system_xstate()?

[    0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
[    0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 0x340 bytes, using 'standard' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.

Re: [all better] Re: regression: massive trouble with fpu rework

[Henrique de Moraes Holschuh]

On Sat, 27 Jun 2015, Mike Galbraith wrote:
> > > BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.
> 
> That BIOS setting is annotated with the helpful text "Disabled for
> Windows XP".  It makes box say interesting things during boot, like...
> 
>         x86/fpu: XSTATE_CPUID missing!
> 
> 
> ..or with HEAD, it triggers warning..
> 
>         if (boot_cpu_data.cpuid_level < XSTATE_CPUID) {
>                 WARN_ON_FPU(1);
>                 return;
>         }
> 
> ..and all kinds of bad juju follows.  I have no idea what the thing does
> beyond what I can interpolate from the word 'limit'.

Well, it is supposed to disable CPUID levels >= 0x04.  This thing should
*NEVER* be enabled, the last operating system that required it to be enabled
was Windows 98.

Can/do we override that crap during cpu init?  If we cannot/don't, maybe
instead of limping along with CPUID crippled, it would be better to either
output a very nasty warning, or outright stop booting [with an appropriate
error message] ?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: [all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Sat, 2015-06-27 at 18:02 -0300, Henrique de Moraes Holschuh wrote:
> On Sat, 27 Jun 2015, Mike Galbraith wrote:
> > > > BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.
> > 
> > That BIOS setting is annotated with the helpful text "Disabled for
> > Windows XP".  It makes box say interesting things during boot, like...
> > 
> >         x86/fpu: XSTATE_CPUID missing!
> > 
> > 
> > ..or with HEAD, it triggers warning..
> > 
> >         if (boot_cpu_data.cpuid_level < XSTATE_CPUID) {
> >                 WARN_ON_FPU(1);
> >                 return;
> >         }
> > 
> > ..and all kinds of bad juju follows.  I have no idea what the thing does
> > beyond what I can interpolate from the word 'limit'.
> 
> Well, it is supposed to disable CPUID levels >= 0x04.  This thing should
> *NEVER* be enabled, the last operating system that required it to be enabled
> was Windows 98.
> 
> Can/do we override that crap during cpu init?  If we cannot/don't, maybe
> instead of limping along with CPUID crippled, it would be better to either
> output a very nasty warning, or outright stop booting [with an appropriate
> error message] ?

Why get all upset?  We didn't even notice before, nor did/does that
other OS.  A casual "BTW, your BIOS sucks.." should suffice, no?

	-Mike

Re: [all better] Re: regression: massive trouble with fpu rework

[Henrique de Moraes Holschuh]

On Sun, 28 Jun 2015, Mike Galbraith wrote:
> > > > > BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.
> > 
> > Well, it is supposed to disable CPUID levels >= 0x04.  This thing should
> > *NEVER* be enabled, the last operating system that required it to be enabled
> > was Windows 98.
> > 
> > Can/do we override that crap during cpu init?  If we cannot/don't, maybe
> > instead of limping along with CPUID crippled, it would be better to either
> > output a very nasty warning, or outright stop booting [with an appropriate
> > error message] ?
> 
> Why get all upset?  We didn't even notice before, nor did/does that
> other OS.  A casual "BTW, your BIOS sucks.." should suffice, no?

Oh, I am not upset, although I suppose my reply did look like it.  Sorry
about that.

It is just that this kind of breakage should not be subtle if we can help
it, because people will use a crippled system for years without noticing...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: [all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Sun, 2015-06-28 at 12:06 -0300, Henrique de Moraes Holschuh wrote:
> On Sun, 28 Jun 2015, Mike Galbraith wrote:
> > > > > > BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.
> > > 
> > > Well, it is supposed to disable CPUID levels >= 0x04.  This thing should
> > > *NEVER* be enabled, the last operating system that required it to be enabled
> > > was Windows 98.
> > > 
> > > Can/do we override that crap during cpu init?  If we cannot/don't, maybe
> > > instead of limping along with CPUID crippled, it would be better to either
> > > output a very nasty warning, or outright stop booting [with an appropriate
> > > error message] ?
> > 
> > Why get all upset?  We didn't even notice before, nor did/does that
> > other OS.  A casual "BTW, your BIOS sucks.." should suffice, no?
> 
> Oh, I am not upset, although I suppose my reply did look like it.  Sorry
> about that.

I didn't mean you personally of course, I meant the kernel ;-)

> It is just that this kind of breakage should not be subtle if we can help
> it, because people will use a crippled system for years without noticing...

If you can use it without noticing for years, it ain't crippled, or?  My
point being that severity seems more akin to the box having a zit behind
its left ear, in which case lobotomizing it seems a tad extreme.

	-Mike

Re: [all better] Re: regression: massive trouble with fpu rework

[Henrique de Moraes Holschuh]

On Sun, 28 Jun 2015, Mike Galbraith wrote:
> On Sun, 2015-06-28 at 12:06 -0300, Henrique de Moraes Holschuh wrote:
> > It is just that this kind of breakage should not be subtle if we can help
> > it, because people will use a crippled system for years without noticing...
> 
> If you can use it without noticing for years, it ain't crippled, or?  My
> point being that severity seems more akin to the box having a zit behind
> its left ear, in which case lobotomizing it seems a tad extreme.

Noted.  However if it does boot with cpuid limited (and we don't "unlimit"
it somehow) on a recent processor, at *best* the user paid good money for a
lot of stuff that is going to not be used to enhance system performance and
system security.

It is not nice to the user to just limp along silently about this.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: [all better] Re: regression: massive trouble with fpu rework

[Ingo Molnar]

* Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> > This would suggest sensitivity on CPUID details, i.e. that doing 
> > fpu__init_system() before other CPU init sequences is causing the bug.
> > 
> > Does the patch below perhaps make a difference? (I'd suggest to apply it 
> > _without_ the other patch I sent.)
> 
> Yup, that made it not care about the BIOS setting.. again.

> > diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> > index 9fc5e3d9d9c8..922c5e0cea4c 100644
> > --- a/arch/x86/kernel/cpu/common.c
> > +++ b/arch/x86/kernel/cpu/common.c
> > @@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
> >  	cpu_detect(c);
> >  	get_cpu_vendor(c);
> >  	get_cpu_cap(c);
> > -	fpu__init_system(c);
> >  
> >  	if (this_cpu->c_early_init)
> >  		this_cpu->c_early_init(c);
> > @@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
> >  		this_cpu->c_bsp_init(c);
> >  
> >  	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
> > +	fpu__init_system(c);
> >  }

Ok, so could you please move the fpu__init_system() further up and see which 
position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

        get_cpu_cap(c);
[0]     fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[1]
        c->cpu_index = 0;
[2]
        filter_cpuid_features(c, false);

[3]
        if (this_cpu->c_bsp_init)
                this_cpu->c_bsp_init(c);

[4]
        setup_force_cpu_cap(X86_FEATURE_ALWAYS);
[5]
}

and we know it from your testing that moving [0] to [5] fixes the crash.

The question is, can we move it to [4], [3], [2] or even [1] instead, without 
breaking the system?

I still don't see where the breakage comes from, but this would help us narrow it 
down.

Thanks,

	Ingo

Re: [all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Mon, 2015-06-29 at 08:40 +0200, Ingo Molnar wrote:
> *
> Ok, so could you please move the fpu__init_system() further up and see which 
> position is that starts breaking with the BIOS option set?
> 
> here's the current, broken layout of the code:
> 
>         get_cpu_cap(c);
> [0]     fpu__init_system(c);
> 
>         if (this_cpu->c_early_init)
>                 this_cpu->c_early_init(c);
> 
> [1]
>         c->cpu_index = 0;
> [2]
>         filter_cpuid_features(c, false);
> 
> [3]
>         if (this_cpu->c_bsp_init)
>                 this_cpu->c_bsp_init(c);
> 
> [4]
>         setup_force_cpu_cap(X86_FEATURE_ALWAYS);
> [5]
> }
> 
> and we know it from your testing that moving [0] to [5] fixes the crash.
> 
> The question is, can we move it to [4], [3], [2] or even [1] instead, without 
> breaking the system?
> 
> I still don't see where the breakage comes from, but this would help us narrow it 
> down.

[0] is the only spot that breaks box.

	-Mike

Re: [all better] Re: regression: massive trouble with fpu rework

[Borislav Petkov]

On Mon, Jun 29, 2015 at 10:25:29AM +0200, Mike Galbraith wrote:
> On Mon, 2015-06-29 at 08:40 +0200, Ingo Molnar wrote:
> > *
> > Ok, so could you please move the fpu__init_system() further up and see which 
> > position is that starts breaking with the BIOS option set?
> > 
> > here's the current, broken layout of the code:
> > 
> >         get_cpu_cap(c);
> > [0]     fpu__init_system(c);
> > 
> >         if (this_cpu->c_early_init)
> >                 this_cpu->c_early_init(c);

> [0] is the only spot that breaks box.

I bet it is that

        /* Unmask CPUID levels if masked: */
        if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
                if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
                                  MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
                        c->cpuid_level = cpuid_eax(0);
                        get_cpu_cap(c);
                }
        }

in early_init_intel(). If you feel like playing, you might comment it
out to see what happens.

:-)

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: [all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Mon, 2015-06-29 at 10:33 +0200, Borislav Petkov wrote:
> On Mon, Jun 29, 2015 at 10:25:29AM +0200, Mike Galbraith wrote:
> > On Mon, 2015-06-29 at 08:40 +0200, Ingo Molnar wrote:
> > > *
> > > Ok, so could you please move the fpu__init_system() further up and see which 
> > > position is that starts breaking with the BIOS option set?
> > > 
> > > here's the current, broken layout of the code:
> > > 
> > >         get_cpu_cap(c);
> > > [0]     fpu__init_system(c);
> > > 
> > >         if (this_cpu->c_early_init)
> > >                 this_cpu->c_early_init(c);
> 
> > [0] is the only spot that breaks box.
> 
> I bet it is that
> 
>         /* Unmask CPUID levels if masked: */
>         if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
>                 if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
>                                   MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
>                         c->cpuid_level = cpuid_eax(0);
>                         get_cpu_cap(c);
>                 }
>         }
> 
> in early_init_intel(). If you feel like playing, you might comment it
> out to see what happens.

I'll poke after I get back from physical tort^Wtherapy.

	-Mike

Re: [all better] Re: regression: massive trouble with fpu rework

[Ingo Molnar]

* Borislav Petkov <bp@alien8.de> wrote:

> On Mon, Jun 29, 2015 at 10:25:29AM +0200, Mike Galbraith wrote:
> > On Mon, 2015-06-29 at 08:40 +0200, Ingo Molnar wrote:
> > > *
> > > Ok, so could you please move the fpu__init_system() further up and see which 
> > > position is that starts breaking with the BIOS option set?
> > > 
> > > here's the current, broken layout of the code:
> > > 
> > >         get_cpu_cap(c);
> > > [0]     fpu__init_system(c);
> > > 
> > >         if (this_cpu->c_early_init)
> > >                 this_cpu->c_early_init(c);
> 
> > [0] is the only spot that breaks box.
> 
> I bet it is that
> 
>         /* Unmask CPUID levels if masked: */
>         if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
>                 if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
>                                   MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
>                         c->cpuid_level = cpuid_eax(0);
>                         get_cpu_cap(c);
>                 }
>         }
> 
> in early_init_intel(). If you feel like playing, you might comment it
> out to see what happens.
> 
> :-)

Indeed, I bet that makes a difference!

I wish that 'unmasking' logic came with more comments:

  - Why do BIOSen ever mask CPUIDs?

  - Why do we unmask the masking?

  - Why doesn't the kernel keep on working just fine even if certain CPUID aspects 
    are turned off?

Thanks,

	Ingo

Re: [all better] Re: regression: massive trouble with fpu rework

[Borislav Petkov]

On Mon, Jun 29, 2015 at 11:35:04AM +0200, Ingo Molnar wrote:
> I wish that 'unmasking' logic came with more comments:
> 
>   - Why do BIOSen ever mask CPUIDs?

Doesn't say a thing why:

066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

SDM doesn't say why either:

"Limit CPUID Maxval (R/W)

When this bit is set to 1, CPUID.00H returns a maximum value in EAX[7:0]
of 3. BIOS should contain a setup question that allows users to specify
when the installed OS does not support CPUID functions greater than 3.

...

Setting this bit may cause unexpected behavior in software that depends
on the availability of CPUID leaves greater than 3."

In the case of hiding XSAVE from windoze ninety-old, probably something
was exploding there, there wasn't a fix to the software so the hardware
had to become soft.

Purely hypothetical, of course.

The last sentence from the SDM quote above also explains why the Linux
workaround of clearing that bit again, exists.

>   - Why do we unmask the masking?

Also purely hypothetical: because Linux doesn't have the windoze
problem.

>   - Why doesn't the kernel keep on working just fine even if certain
>   CPUID aspects are turned off?

I guess that should be doable but one has to get such a box, enable
that BIOS feature and fix all the fallout that happens. Provided it
can be fixed. hpa went and reenabled those CPUID leafs instead in
066941bd4eeb1.

I guess we simply shouldn't do any CPUID-dependent stuff before:

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

and slap a big fat comment above it.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: [all better] Re: regression: massive trouble with fpu rework

[Mike Galbraith]

On Mon, 2015-06-29 at 10:33 +0200, Borislav Petkov wrote:

> I bet it is that
> 
>         /* Unmask CPUID levels if masked: */
>         if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
>                 if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
>                                   MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
>                         c->cpuid_level = cpuid_eax(0);
>                         get_cpu_cap(c);
>                 }
>         }
> 
> in early_init_intel(). If you feel like playing, you might comment it
> out to see what happens.

With it commented out, and fpu__init_system() either back at previously
booting position [5] or at original [0], doesn't matter, box is dead,
but differently.  It stalls after setting clocksource to tsc, and just
sits there.

	-Mike

Re: [all better] Re: regression: massive trouble with fpu rework

[Borislav Petkov]

On Mon, Jun 29, 2015 at 02:27:23PM +0200, Mike Galbraith wrote:
> With it commented out, and fpu__init_system() either back at previously
> booting position [5] or at original [0], doesn't matter, box is dead,
> but differently.  It stalls after setting clocksource to tsc, and just
> sits there.

... which means that unmasking the CPUID features is absolutely needed
on Linux. Not unmasking probably triggers this original bug which

  066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

fixed.

-- 
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--

Re: [all better] Re: regression: massive trouble with fpu rework

[H. Peter Anvin]

On 06/29/2015 02:35 AM, Ingo Molnar wrote:
> 
> Indeed, I bet that makes a difference!
> 
> I wish that 'unmasking' logic came with more comments:
> 
>   - Why do BIOSen ever mask CPUIDs?
> 

To work around bugs in legacy operating systems.

>   - Why do we unmask the masking?

Because we don't have those specific bugs.

>   - Why doesn't the kernel keep on working just fine even if certain CPUID aspects 
>     are turned off?

Because it exercises code paths that are otherwise impossible, for
example, it exposes the XSAVE capability without exposing the XSAVE
information in higher CPUID leaves.

The other option would be to have a list of CPU features that should be
turned off whenever the CPUID leaf maximum is too low, but it gives a
better user experience to just override the BIOS capping and then we
have fewer code paths in the kernel to worry about.

	-hpa

Re: [all better] Re: regression: massive trouble with fpu rework

[H. Peter Anvin]

On 06/28/2015 11:40 PM, Ingo Molnar wrote:
> 
> Ok, so could you please move the fpu__init_system() further up and see which 
> position is that starts breaking with the BIOS option set?
> 
> here's the current, broken layout of the code:
> 
>         get_cpu_cap(c);
> [0]     fpu__init_system(c);
> 
>         if (this_cpu->c_early_init)
>                 this_cpu->c_early_init(c);
> 
> [1]
>         c->cpu_index = 0;
> [2]
>         filter_cpuid_features(c, false);
> 
> [3]
>         if (this_cpu->c_bsp_init)
>                 this_cpu->c_bsp_init(c);
> 
> [4]
>         setup_force_cpu_cap(X86_FEATURE_ALWAYS);
> [5]
> }
> 
> and we know it from your testing that moving [0] to [5] fixes the crash.
> 
> The question is, can we move it to [4], [3], [2] or even [1] instead, without 
> breaking the system?
> 
> I still don't see where the breakage comes from, but this would help us narrow it 
> down.
> 

It should be moved to [4] or [5].  I would argue that the line setting
X86_FEATURE_ALWAYS should moved up and then fpu__init_system(c) should
be moved after the c_bsp_init() line.

	-hpa

Re: [all better] Re: regression: massive trouble with fpu rework

[Ingo Molnar]

* H. Peter Anvin <hpa@zytor.com> wrote:

> On 06/29/2015 02:35 AM, Ingo Molnar wrote:
> > 
> > Indeed, I bet that makes a difference!
> > 
> > I wish that 'unmasking' logic came with more comments:
> > 
> >   - Why do BIOSen ever mask CPUIDs?
> > 
> 
> To work around bugs in legacy operating systems.
>
> >   - Why do we unmask the masking?
> 
> Because we don't have those specific bugs.

Great - would be nice to put those reasons between /* */ markers, to keep future 
generations (and overworked maintainers!) from wondering.

> >   - Why doesn't the kernel keep on working just fine even if certain CPUID aspects 
> >     are turned off?
> 
> Because it exercises code paths that are otherwise impossible, for example, it 
> exposes the XSAVE capability without exposing the XSAVE information in higher 
> CPUID leaves.
> 
> The other option would be to have a list of CPU features that should be turned 
> off whenever the CPUID leaf maximum is too low, but it gives a better user 
> experience to just override the BIOS capping and then we have fewer code paths 
> in the kernel to worry about.

1)

As a side note, I think we should generally be robust enough to recognize pretty 
much any CPUID 'mischief' and at minimum not crash.

2)

But this FPU crash is different, here the reason for the crash is the following 
bug in the FPU code:

	fpu__init_system(); /* inits the FPU based on masked CPUID */

	... CPUID *extends* ...

	fpu__init_cpu(); /* Actually uses the FPU now based on the expanded CPUID */

        *KABOOM*

I.e. we (obviously) should not base half on the FPU logic on different CPUID bits 
than the other half of the FPU logic.

I'll queue up the fix, which is to do the early FPU init after our CPUID state 
stabilizes. (i.e. the second patch I sent to Mike.)

Thanks,

	Ingo

Re: [all better] Re: regression: massive trouble with fpu rework

[Ingo Molnar]

* Borislav Petkov <bp@alien8.de> wrote:

> On Mon, Jun 29, 2015 at 02:27:23PM +0200, Mike Galbraith wrote:
> > With it commented out, and fpu__init_system() either back at previously
> > booting position [5] or at original [0], doesn't matter, box is dead,
> > but differently.  It stalls after setting clocksource to tsc, and just
> > sits there.
> 
> ... which means that unmasking the CPUID features is absolutely needed
> on Linux. Not unmasking probably triggers this original bug which
> 
>   066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")
> 
> fixed.

Yes.

And I'd consider us hanging a separate (but not high prio) bug: the kernel should 
be robust as long as the CPUID data is stable. In that sense the original fix is 
right (we really want to unmask all available CPUID leaves), but it also masked 
another (less severe) kernel bug.

For example virtualization is known to tweak CPUID details creatively, and 
firmware (as this example shows it) can mess it up a well, so we generally want to 
treat it as untrusted input data that needs to be validated.

Thanks,

	Ingo

Re: [all better] Re: regression: massive trouble with fpu rework

[Ingo Molnar]

* H. Peter Anvin <hpa@zytor.com> wrote:

> On 06/28/2015 11:40 PM, Ingo Molnar wrote:
> > 
> > Ok, so could you please move the fpu__init_system() further up and see which 
> > position is that starts breaking with the BIOS option set?
> > 
> > here's the current, broken layout of the code:
> > 
> >         get_cpu_cap(c);
> > [0]     fpu__init_system(c);
> > 
> >         if (this_cpu->c_early_init)
> >                 this_cpu->c_early_init(c);
> > 
> > [1]
> >         c->cpu_index = 0;
> > [2]
> >         filter_cpuid_features(c, false);
> > 
> > [3]
> >         if (this_cpu->c_bsp_init)
> >                 this_cpu->c_bsp_init(c);
> > 
> > [4]
> >         setup_force_cpu_cap(X86_FEATURE_ALWAYS);
> > [5]
> > }
> > 
> > and we know it from your testing that moving [0] to [5] fixes the crash.
> > 
> > The question is, can we move it to [4], [3], [2] or even [1] instead, without 
> > breaking the system?
> > 
> > I still don't see where the breakage comes from, but this would help us narrow it 
> > down.
> > 
> 
> It should be moved to [4] or [5].  I would argue that the line setting
> X86_FEATURE_ALWAYS should moved up and then fpu__init_system(c) should
> be moved after the c_bsp_init() line.

Yeah, so the patch I sent to Mike (and which solved the bug) moved it to [5].

Thanks,

	Ingo

[tip:x86/urgent] x86/fpu: Fix FPU related boot regression when CPUID masking BIOS feature is enabled

[tip-bot for Ingo Molnar]

Commit-ID:  db52ef74b35dcb91fd154fa52c618bdd1b90e28e
Gitweb:     http://git.kernel.org/tip/db52ef74b35dcb91fd154fa52c618bdd1b90e28e
Author:     Ingo Molnar <mingo@kernel.org>
AuthorDate: Sat, 27 Jun 2015 10:25:14 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 30 Jun 2015 07:22:10 +0200

x86/fpu: Fix FPU related boot regression when CPUID masking BIOS feature is enabled

Mike Galbraith reported:

  " My i7-4790 box is having one hell of a time with this merge
    window, dead in the water.

    BIOS setting "Limit CPUID Maximum" upsets new fpu code
    mightily. "

It turns out that Linux does a double workaround here, as per:

  066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

it undoes the BIOS workaround - but as a side effect the CPUID
state is not completely constant during early init anymore,
and the new FPU init code did not take this into account.

So what happened is that the xstate init code did not have full
CPUID available, which broke subsequent attempts to use xstate
features.

Fix this by ordering the early FPU init code to after we've
stabilized the CPUID state.

Reported-bisected-and-tested-by: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20150627082514.GA10894@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 9fc5e3d..922c5e0 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 	cpu_detect(c);
 	get_cpu_vendor(c);
 	get_cpu_cap(c);
-	fpu__init_system(c);
 
 	if (this_cpu->c_early_init)
 		this_cpu->c_early_init(c);
@@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		this_cpu->c_bsp_init(c);
 
 	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
+	fpu__init_system(c);
 }
 
 void __init early_cpu_init(void)

Re: [all better] Re: regression: massive trouble with fpu rework

[H. Peter Anvin]

On 06/29/2015 10:16 PM, Ingo Molnar wrote:
> 
> * Borislav Petkov <bp@alien8.de> wrote:
> 
>> On Mon, Jun 29, 2015 at 02:27:23PM +0200, Mike Galbraith wrote:
>>> With it commented out, and fpu__init_system() either back at previously
>>> booting position [5] or at original [0], doesn't matter, box is dead,
>>> but differently.  It stalls after setting clocksource to tsc, and just
>>> sits there.
>>
>> ... which means that unmasking the CPUID features is absolutely needed
>> on Linux. Not unmasking probably triggers this original bug which
>>
>>   066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")
>>
>> fixed.
> 
> Yes.
> 
> And I'd consider us hanging a separate (but not high prio) bug: the kernel should 
> be robust as long as the CPUID data is stable. In that sense the original fix is 
> right (we really want to unmask all available CPUID leaves), but it also masked 
> another (less severe) kernel bug.
> 
> For example virtualization is known to tweak CPUID details creatively, and 
> firmware (as this example shows it) can mess it up a well, so we generally want to 
> treat it as untrusted input data that needs to be validated.
> 

Well, that is not *entirely* possible, since if the data is just plain
wrong, we're screwed no matter what.

However, we could deal with CPUID level capping.  The best way to do
that is probably to have a table of CPU features and the minimum
required CPUID level for each.  If maximum CPUID level < that level,
disable that feature.

	-hpa

Re: [all better] Re: regression: massive trouble with fpu rework

[Henrique de Moraes Holschuh]

On Tue, 30 Jun 2015, Ingo Molnar wrote:
> And I'd consider us hanging a separate (but not high prio) bug: the kernel should 
> be robust as long as the CPUID data is stable. In that sense the original fix is 
> right (we really want to unmask all available CPUID leaves), but it also masked 
> another (less severe) kernel bug.
> 
> For example virtualization is known to tweak CPUID details creatively, and 
> firmware (as this example shows it) can mess it up a well, so we generally want to 
> treat it as untrusted input data that needs to be validated.

Processor microcode updates can also change cpuid information, at least on
Intel.  There are Intel microcode updates in the field that do this.

Specific Intel MSR writes *should* be able to change cpuid information as
well, as they enable/disable features that are reflected by a cpuid bit.

I have no idea about AMD, though.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh