From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PULL_REQUEST,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0BEEC10F00 for ; Thu, 21 Mar 2019 20:26:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8D43D218D3 for ; Thu, 21 Mar 2019 20:26:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728930AbfCUU0r (ORCPT ); Thu, 21 Mar 2019 16:26:47 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:39305 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728566AbfCUU0q (ORCPT ); Thu, 21 Mar 2019 16:26:46 -0400 Received: from localhost ([127.0.0.1] helo=flow.W.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1h74Gr-00057y-MT; Thu, 21 Mar 2019 21:26:37 +0100 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen Subject: [PATCH v8] x86: load FPU registers on return to userland Date: Thu, 21 Mar 2019 21:26:08 +0100 Message-Id: <20190321202632.16810-1-bigeasy@linutronix.de> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This is a refurbished series originally started by by Rik van Riel. The goal is load the FPU registers on return to userland and not on every context switch. By this optimisation we can: - avoid loading the registers if the task stays in kernel and does not return to userland - make kernel_fpu_begin() cheaper: it only saves the registers on the first invocation. The second invocation does not need save them again. To access the FPU registers in kernel we need: - disable preemption to avoid that the scheduler switches tasks. By doing so it would set TIF_NEED_FPU_LOAD and the FPU registers would be not valid. - disable BH because the softirq might use kernel_fpu_begin() and then set TIF_NEED_FPU_LOAD instead loading the FPU registers on completion. v7…v8: Rebased on top of v5.1-rc1. And then: - Remove a WARN_ON() in switch_fpu_finish. Turns out it can trigger during a preemption while the xstate is initialized. - Provide __read_pkru_ins() and __write_pkru_ins which are symmetrical. __write_pkru() does the "write only if the value is different from current" check. - The kernel threads now load `init_pkru_value' instead of `0' for the PKRU value. - The PKRU value is also written into its xsave area of init_fpstate. v6…v7: Rebased on top of v5.0-rc7 and addressed Borislav Petkov's review. v5…v6: Rebased on top of v5.0-rc1. Integrated a few fixes which I noticed while looking over the patches, dropped the first few patches which were already applied. v4…v5: Rebased on top of a fix, noticed a problem with XSAVES and then redid the restore on sig return (patch #26 to #28). I don't like very much the sig save+restore thing that we are doing. It has been always like that. I *think* that this is just because we have nowhere to stash the FPU state while we are handling the signal. We could add another fpu->state for the signal handler and avoid the thing. Debian code-search revealed that `criu' is using it (and I didn't figure out why). Nothing else (that is packaged in Debian). Maybe we could get rid of this and if `criu' would then use a dedicated interface for its needs rather the signal interface that happen to do what it wants :) v3…v4: It has been suggested to remove the `initialized' member of the struct fpu because it should not required be needed with lazy-FPU-restore and would make the review easier. This is the first part of the series, the second is basically the rebase of the v3 queue. As a result, the diffstat became negative (which wasn't the case in previous version) :) I tried to incorporate all the review comments that came up, some of them were "outdated" after the removal of the `initialized' member. I'm sorry should I missed any. v1…v3: v2 was never posted. I followed the idea to completely decouple PKRU from xstate. This didn't quite work and made a few things complicated. One obvious required fixup is copy_fpstate_to_sigframe() where the PKRU state needs to be fiddled into xstate. This required another xfeatures_mask so that the sanity checks were performed and xstate_offsets would be computed. Additionally ptrace also reads/sets xstate in order to get/set the register and PKRU is one of them. So this would need some fiddle, too. In v3 I dropped that decouple idea. I also learned that the wrpkru instruction is not privileged and so caching it in kernel does not work. Instead I keep PKRU in xstate area and load it at context switch time while the remaining registers are deferred (until return to userland). The offset of PKRU within xstate is enumerated at boot time so why not use it. The following changes since commit 9e98c678c2d6ae3a17cb2de55d17f69dddaa231b: Linux 5.1-rc1 (2019-03-17 14:22:26 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/bigeasy/staging.git x86_fpu_rtu_v8 for you to fetch changes up to 7c9f11507fce5906ae0b475240fdba80b3c76a0a: x86/pkeys: add PKRU value to init_fpstate (2019-03-21 21:22:45 +0100) ---------------------------------------------------------------- Rik van Riel (5): x86/fpu: Add (__)make_fpregs_active helpers x86/fpu: Eager switch PKRU state x86/fpu: Always store the registers in copy_fpstate_to_sigframe() x86/fpu: Prepare copy_fpstate_to_sigframe() for TIF_NEED_FPU_LOAD x86/fpu: Defer FPU state load until return to userspace Sebastian Andrzej Siewior (19): x86/fpu: Remove fpu->initialized usage in __fpu__restore_sig() x86/fpu: Remove fpu__restore() x86/fpu: Remove preempt_disable() in fpu__clear() x86/fpu: Always init the `state' in fpu__clear() x86/fpu: Remove fpu->initialized usage in copy_fpstate_to_sigframe() x86/fpu: Don't save fxregs for ia32 frames in copy_fpstate_to_sigframe() x86/fpu: Remove fpu->initialized x86/fpu: Remove user_fpu_begin() x86/fpu: Make __raw_xsave_addr() use feature number instead of mask x86/fpu: Make get_xsave_field_ptr() and get_xsave_addr() use feature number instead of mask x86/pkru: Provide .*_pkru_ins() functions x86/fpu: Only write PKRU if it is different from current x86/pkeys: Don't check if PKRU is zero before writting it x86/entry: Add TIF_NEED_FPU_LOAD x86/fpu: Update xstate's PKRU value on write_pkru() x86/fpu: Inline copy_user_to_fpregs_zeroing() x86/fpu: Let __fpu__restore_sig() restore the !32bit+fxsr frame from kernel memory x86/fpu: Merge the two code paths in __fpu__restore_sig() x86/pkeys: add PKRU value to init_fpstate Documentation/preempt-locking.txt | 1 - arch/x86/entry/common.c | 8 ++ arch/x86/ia32/ia32_signal.c | 17 ++- arch/x86/include/asm/fpu/api.h | 31 ++++++ arch/x86/include/asm/fpu/internal.h | 163 ++++++++++++++--------------- arch/x86/include/asm/fpu/signal.h | 2 +- arch/x86/include/asm/fpu/types.h | 9 -- arch/x86/include/asm/fpu/xstate.h | 5 +- arch/x86/include/asm/pgtable.h | 28 ++++- arch/x86/include/asm/special_insns.h | 18 +++- arch/x86/include/asm/thread_info.h | 2 + arch/x86/include/asm/trace/fpu.h | 8 +- arch/x86/kernel/cpu/common.c | 5 + arch/x86/kernel/fpu/core.c | 194 +++++++++++++++++------------------ arch/x86/kernel/fpu/init.c | 2 - arch/x86/kernel/fpu/regset.c | 24 ++--- arch/x86/kernel/fpu/signal.c | 191 +++++++++++++++------------------- arch/x86/kernel/fpu/xstate.c | 42 ++++---- arch/x86/kernel/process.c | 2 +- arch/x86/kernel/process_32.c | 11 +- arch/x86/kernel/process_64.c | 11 +- arch/x86/kernel/signal.c | 17 ++- arch/x86/kernel/traps.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/x86.c | 48 +++++---- arch/x86/math-emu/fpu_entry.c | 3 - arch/x86/mm/mpx.c | 6 +- arch/x86/mm/pkeys.c | 21 ++-- 28 files changed, 439 insertions(+), 434 deletions(-) Sebastian