Re: [PATCH v5 00/34] x86: enable FRED for x86-64

["Kang, Shan" <shan.kang@intel.com>]

We wanted to check whether there are KVM regressions with the v5 FRED patch set,
and had a round of Kselftest on KVM guests. 
Following are the results on X86-64.

+--------------------------------+-------+-------+-------+-------+
|             Config             |  Pass |  Fail |  Skip |  Hang |
+--------------------------------+-------+-------+-------+-------+
| the 7th Intel(R) Core(TM) CPU  |       |       |       |       |
|   | host 6.3.0-rc1+ w/ FRED    |  2720 |  403  |  689  |   9   |
|   patch set FRED disabled |    |       |       |       |       |
|        guest 6.3.0-rc1+        |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
| the 7th Intel(R) Core(TM) CPU  |       |       |       |       |
|   | host 6.3.0-rc1+ w/ FRED    |  2720 |  403  |  689  |   9   |
|  patch set | guest 6.3.0-rc1+  |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|  Intel Simics® Simulator w/o   |       |       |       |       |
| FRED model | host 6.3.0-rc1+ | |  1403 |  277  |  2127 |   14  |
|        guest 6.3.0-rc1+        |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|  Intel Simics® Simulator w/o   |       |       |       |       |
|  FRED model | host 6.3.0-rc1+  |       |       |       |       |
|   w/ FRED patch set | guest    |  1403 |  277  |  2127 |   14  |
|  6.3.0-rc1+ w/ FRED patch set  |       |       |       |       |
|         FRED disabled          |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|  Intel Simics® Simulator w/o   |       |       |       |       |
|  FRED model | host 6.3.0-rc1+  |  1403 |  277  |  2127 |   14  |
|   w/ FRED patch set | guest    |       |       |       |       |
|  6.3.0-rc1+ w/ FRED patch set  |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|   Intel Simics® Simulator w/   |       |       |       |       |
| FRED model + host 6.3.0-rc1+ | |  1404 |  276  |  2127 |   14  |
| guest 6.3.0-rc1+ w/ FRED patch |       |       |       |       |
|              set               |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|   Intel Simics® Simulator w/   |       |       |       |       |
|  FRED model | host 6.3.0-rc1+  |  1404 |  276  |  2127 |   14  |
|     w/ FRED patch set FRED     |       |       |       |       |
|  disabled | guest 6.3.0-rc1+   |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|   Intel Simics® Simulator w/   |       |       |       |       |
|  FRED model | host 6.3.0-rc1+  |       |       |       |       |
|     w/ FRED patch set FRED     |  1404 |  276  |  2127 |   14  |
| disabled | guest 6.3.0-rc1+ w/ |       |       |       |       |
|         FRED patch set         |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|   Intel Simics® Simulator w/   |       |       |       |       |
|  FRED model | host 6.3.0-rc1+  |  1404 |  276  |  2127 |   14  |
|   w/ FRED patch set | guest    |       |       |       |       |
|           6.3.0-rc1+           |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|   Intel Simics® Simulator w/   |       |       |       |       |
|  FRED model | host 6.3.0-rc1+  |       |       |       |       |
|   w/ FRED patch set | guest    |  1404 |  276  |  2127 |   14  |
|  6.3.0-rc1+ w/ FRED patch set  |       |       |       |       |
|         FRED disabled          |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+
|   Intel Simics® Simulator w/   |       |       |       |       |
|  FRED model | host 6.3.0-rc1+  |  1404 |  276  |  2127 |   14  |
|   w/ FRED patch set | guest    |       |       |       |       |
|  6.3.0-rc1+ w/ FRED patch set  |       |       |       |       |
+--------------------------------+-------+-------+-------+-------+

The Simics FRED model has one more passed case, which is x86:test_vsyscall_32,
due to https://lore.kernel.org/lkml/20230311084824.2340-1-xin3.li@intel.com/.

Thanks
   --Shan

On Sat, 2023-03-11 at 09:58 +0000, Kang, Shan wrote:
> We tested the v5 FRED patch set on the Intel Simics® Simulator and a machine
> with a 7th Intel(R) Core(TM) CPU.
> 
> Following are the Kselftest results on X86-64.
> +--------------------------------------------+-------+-------+-------+-------+
> >                  Config                    |  Pass |  Fail |  Skip |  Hang |
> +--------------------------------------------+-------+-------+-------+-------+
> >       the 7th Intel(R) Core(TM) CPU        |  3078 |  458  |  734  |   5   |
> >                 6.3.0-rc1+                 |       |       |       |       |
> +--------------------------------------------+-------+-------+-------+-------+
> >       the 7th Intel(R) Core(TM) CPU        |  3078 |  458  |  734  |   5   |
> >        6.3.0-rc1+ w/ FRED patch set        |       |       |       |       |
> +--------------------------------------------+-------+-------+-------+-------+
> >   Intel Simics® Simulator w/o FRED model   |  1888 |  271  |  2105 |   11  |
> >                 6.3.0-rc1+                 |       |       |       |       |
> +--------------------------------------------+-------+-------+-------+-------+
> >   Intel Simics® Simulator w/o FRED model   |  1888 |  271  |  2105 |   11  |
> >        6.3.0-rc1+ w/ FRED patch set        |       |       |       |       |
> +--------------------------------------------+-------+-------+-------+-------+
> >   Intel Simics® Simulator w/ FRED model    |  1889 |  270  |  2105 |   11  |
> >                 6.3.0-rc1+                 |       |       |       |       |
> +--------------------------------------------+-------+-------+-------+-------+
> >   Intel Simics® Simulator w/ FRED model    |  1889 |  270  |  2105 |   11  |
> > 6.3.0-rc1+ w/ FRED patch set FRED disabled |       |       |       |       |
> +--------------------------------------------+-------+-------+-------+-------+
> >   Intel Simics® Simulator w/ FRED model    |  1888 |  270  |  2105 |   12  |
> >        6.3.0-rc1+ w/ FRED patch set        |       |       |       |       |
> +--------------------------------------------+-------+-------+-------+-------+
> 
> The following issues are seen in this round of test.
> +----------------+----------------+----------------+----------------+
> >                | x86:test_      | bpf:test_progs | x86:sysret     |
> >                | vsyscall_32    |                |    _rip_64     |
> +----------------+----------------+----------------+----------------+
> >    the 7th     |                |                |                |
> >    Intel(R)    |      FAIL      |      FAIL      |      PASS      |
> >  Core(TM) CPU  |                |                |                |
> >   6.3.0-rc1+   |                |                |                |
> +----------------+----------------+----------------+----------------+
> >    the 7th     |                |                |                |
> >    Intel(R)    |                |                |                |
> >  Core(TM) CPU  |      FAIL      |      FAIL      |      PASS      |
> > 6.3.0-rc1+ w/  |                |                |                |
> > FRED patch set |                |                |                |
> +----------------+----------------+----------------+----------------+
> > Intel Simics®  |                |                |                |
> > Simulator w/o  |      FAIL      |      FAIL      |      PASS      |
> >   FRED model   |                |                |                |
> >   6.3.0-rc1+   |                |                |                |
> +----------------+----------------+----------------+----------------+
> > Intel Simics®  |                |                |                |
> > Simulator w/o  |                |                |                |
> >   FRED model   |      FAIL      |      FAIL      |      PASS      |
> > 6.3.0-rc1+ w/  |                |                |                |
> > FRED patch set |                |                |                |
> +----------------+----------------+----------------+----------------+
> > Intel Simics®  |                |                |                |
> >  Simulator w/  |      PASS      |      FAIL      |      PASS      |
> >   FRED model   |                |                |                |
> >   6.3.0-rc1+   |                |                |                |
> +----------------+----------------+----------------+----------------+
> > Intel Simics®  |                |                |                |
> >  Simulator w/  |                |                |                |
> >   FRED model   |      PASS      |      FAIL      |      PASS      |
> > 6.3.0-rc1+ w/  |                |                |                |
> > FRED patch set |                |                |                |
> > FRED disabled  |                |                |                |
> +----------------+----------------+----------------+----------------+
> > Intel Simics®  |                |                |                |
> >  Simulator w/  |                |                |                |
> >   FRED model   |      PASS      |      HANG      |      FAIL      |
> > 6.3.0-rc1+ w/  |                |                |                |
> > FRED patch set |                |                |                |
> +----------------+----------------+----------------+----------------+
> 
> The test "x86:sysret_rip_64" is NOT a valid test on FRED, and there is a fix
> from Ammar Faizi after we discussed it in the LKML.
> 
> The test "bpf:test_progs" is still in investigation.
> 
> The "x86:test_vsyscall_32" is a regression since the v3 FRED patch set.
> 
> Thanks
>    --Shan
> 
> On Mon, 2023-03-06 at 18:39 -0800, Xin Li wrote:
> > This patch set enables FRED for x86-64.
> > 
> > The Intel flexible return and event delivery (FRED) architecture defines
> > simple
> > new transitions that change privilege level (ring transitions). The FRED
> > architecture was designed with the following goals:
> > 1) Improve overall performance and response time by replacing event delivery
> > through the interrupt descriptor table (IDT event delivery) and event return
> > by
> > the IRET instruction with lower latency transitions.
> > 2) Improve software robustness by ensuring that event delivery establishes
> > the
> > full supervisor context and that event return establishes the full user
> > context.
> > 
> > The new transitions defined by the FRED architecture are FRED event delivery
> > and,
> > for returning from events, two FRED return instructions. FRED event delivery
> > can
> > effect a transition from ring 3 to ring 0, but it is used also to deliver
> > events
> > incident to ring 0. One FRED instruction (ERETU) effects a return from ring
> > 0
> > to
> > ring 3, while the other (ERETS) returns while remaining in ring 0.
> > 
> > Search for the latest FRED spec in most search engines with this search
> > pattern:
> > 
> >   site:intel.com FRED (flexible return and event delivery) specification
> > 
> > As of now there is no publicly avaiable CPU supporting FRED, thus the Intel
> > Simics® Simulator is used as software development and testing vehicles. And
> > it can be downloaded from:
> >   
> > https://www.intel.com/content/www/us/en/developer/articles/tool/simics-simulator.html
> > 
> > To enable FRED, the Simics package 8112 QSP-CPU needs to be installed with
> > CPU
> > model configured as:
> > 	$cpu_comp_class = "x86-experimental-fred"
> > 
> > Longer term, we should refactor common code shared by FRED and IDT into
> > common
> > shared files, and contain IDT code using a new config CONFIG_X86_IDT.
> > 
> > Changes since v4:
> > * Rebased against v6.3-rc1.
> > * Do NOT use the term "injection", which in the KVM context means to
> >   reinject an event into the guest (Sean Christopherson).
> > * Add the explanation of why to execute "int $2" to invoke the NMI handler
> >   in NMI caused VM exits (Sean Christopherson).
> > * Use cs/ss instead of csx/ssx when initializing the pt_regs structure
> >   for calling external_interrupt(), otherwise it breaks i386 build.
> > 
> > Changes since v3:
> > * Call external_interrupt() to handle IRQ in IRQ caused VM exits.
> > * Execute "int $2" to handle NMI in NMI caused VM exits.
> > * Rename csl/ssl of the pt_regs structure to csx/ssx (x for extended)
> >   (Andrew Cooper).
> > 
> > Changes since v2:
> > * Improve comments for changes in arch/x86/include/asm/idtentry.h.
> > 
> > Changes since v1:
> > * call irqentry_nmi_{enter,exit}() in both IDT and FRED debug fault kernel
> >   handler (Peter Zijlstra).
> > * Initialize a FRED exception handler to fred_bad_event() instead of NULL
> >   if no FRED handler defined for an exception vector (Peter Zijlstra).
> > * Push calling irqentry_{enter,exit}() and instrumentation_{begin,end}()
> >   down into individual FRED exception handlers, instead of in the dispatch
> >   framework (Peter Zijlstra).
> > 
> > 
> > H. Peter Anvin (Intel) (24):
> >   x86/traps: let common_interrupt() handle IRQ_MOVE_CLEANUP_VECTOR
> >   x86/traps: add a system interrupt table for system interrupt dispatch
> >   x86/traps: add external_interrupt() to dispatch external interrupts
> >   x86/cpufeature: add the cpu feature bit for FRED
> >   x86/opcode: add ERETU, ERETS instructions to x86-opcode-map
> >   x86/objtool: teach objtool about ERETU and ERETS
> >   x86/cpu: add X86_CR4_FRED macro
> >   x86/fred: add Kconfig option for FRED (CONFIG_X86_FRED)
> >   x86/fred: if CONFIG_X86_FRED is disabled, disable FRED support
> >   x86/cpu: add MSR numbers for FRED configuration
> >   x86/fred: header file with FRED definitions
> >   x86/fred: make unions for the cs and ss fields in struct pt_regs
> >   x86/fred: reserve space for the FRED stack frame
> >   x86/fred: add a page fault entry stub for FRED
> >   x86/fred: add a debug fault entry stub for FRED
> >   x86/fred: add a NMI entry stub for FRED
> >   x86/fred: FRED entry/exit and dispatch code
> >   x86/fred: FRED initialization code
> >   x86/fred: update MSR_IA32_FRED_RSP0 during task switch
> >   x86/fred: let ret_from_fork() jmp to fred_exit_user when FRED is
> >     enabled
> >   x86/fred: disallow the swapgs instruction when FRED is enabled
> >   x86/fred: no ESPFIX needed when FRED is enabled
> >   x86/fred: allow single-step trap and NMI when starting a new thread
> >   x86/fred: allow FRED systems to use interrupt vectors 0x10-0x1f
> > 
> > Xin Li (10):
> >   x86/traps: add install_system_interrupt_handler()
> >   x86/traps: export external_interrupt() for VMX IRQ reinjection
> >   x86/fred: header file for event types
> >   x86/fred: add a machine check entry stub for FRED
> >   x86/fred: fixup fault on ERETU by jumping to fred_entrypoint_user
> >   x86/ia32: do not modify the DPL bits for a null selector
> >   x86/fred: allow dynamic stack frame size
> >   x86/fred: disable FRED by default in its early stage
> >   KVM: x86/vmx: call external_interrupt() to handle IRQ in IRQ caused VM
> >     exits
> >   KVM: x86/vmx: execute "int $2" to handle NMI in NMI caused VM exits
> >     when FRED is enabled
> > 
> >  .../admin-guide/kernel-parameters.txt         |   4 +
> >  arch/x86/Kconfig                              |   9 +
> >  arch/x86/entry/Makefile                       |   5 +-
> >  arch/x86/entry/entry_32.S                     |   2 +-
> >  arch/x86/entry/entry_64.S                     |   5 +
> >  arch/x86/entry/entry_64_fred.S                |  59 +++++
> >  arch/x86/entry/entry_fred.c                   | 234 ++++++++++++++++++
> >  arch/x86/entry/vsyscall/vsyscall_64.c         |   2 +-
> >  arch/x86/include/asm/cpufeatures.h            |   1 +
> >  arch/x86/include/asm/disabled-features.h      |   8 +-
> >  arch/x86/include/asm/entry-common.h           |   3 +
> >  arch/x86/include/asm/event-type.h             |  17 ++
> >  arch/x86/include/asm/extable_fixup_types.h    |   4 +-
> >  arch/x86/include/asm/fred.h                   | 131 ++++++++++
> >  arch/x86/include/asm/idtentry.h               |  76 +++++-
> >  arch/x86/include/asm/irq.h                    |   5 +
> >  arch/x86/include/asm/irq_vectors.h            |  15 +-
> >  arch/x86/include/asm/msr-index.h              |  13 +-
> >  arch/x86/include/asm/processor.h              |  12 +-
> >  arch/x86/include/asm/ptrace.h                 |  36 ++-
> >  arch/x86/include/asm/switch_to.h              |  10 +-
> >  arch/x86/include/asm/thread_info.h            |  35 +--
> >  arch/x86/include/asm/traps.h                  |  13 +
> >  arch/x86/include/asm/vmx.h                    |  17 +-
> >  arch/x86/include/uapi/asm/processor-flags.h   |   2 +
> >  arch/x86/kernel/Makefile                      |   1 +
> >  arch/x86/kernel/apic/apic.c                   |  11 +-
> >  arch/x86/kernel/apic/vector.c                 |   8 +-
> >  arch/x86/kernel/cpu/acrn.c                    |   7 +-
> >  arch/x86/kernel/cpu/common.c                  |  88 ++++---
> >  arch/x86/kernel/cpu/mce/core.c                |  11 +
> >  arch/x86/kernel/cpu/mshyperv.c                |  22 +-
> >  arch/x86/kernel/espfix_64.c                   |   8 +
> >  arch/x86/kernel/fred.c                        |  73 ++++++
> >  arch/x86/kernel/head_32.S                     |   3 +-
> >  arch/x86/kernel/idt.c                         |   6 +-
> >  arch/x86/kernel/irq.c                         |   6 +-
> >  arch/x86/kernel/irqinit.c                     |   7 +-
> >  arch/x86/kernel/kvm.c                         |   4 +-
> >  arch/x86/kernel/nmi.c                         |  28 +++
> >  arch/x86/kernel/process.c                     |   5 +
> >  arch/x86/kernel/process_64.c                  |  21 +-
> >  arch/x86/kernel/signal_32.c                   |  21 +-
> >  arch/x86/kernel/traps.c                       | 175 +++++++++++--
> >  arch/x86/kvm/vmx/vmx.c                        |  33 ++-
> >  arch/x86/lib/x86-opcode-map.txt               |   2 +-
> >  arch/x86/mm/extable.c                         |  28 +++
> >  arch/x86/mm/fault.c                           |  20 +-
> >  drivers/xen/events/events_base.c              |   5 +-
> >  kernel/fork.c                                 |   6 +
> >  tools/arch/x86/include/asm/cpufeatures.h      |   1 +
> >  .../arch/x86/include/asm/disabled-features.h  |   8 +-
> >  tools/arch/x86/include/asm/msr-index.h        |  13 +-
> >  tools/arch/x86/lib/x86-opcode-map.txt         |   2 +-
> >  tools/objtool/arch/x86/decode.c               |  19 +-
> >  55 files changed, 1185 insertions(+), 175 deletions(-)
> >  create mode 100644 arch/x86/entry/entry_64_fred.S
> >  create mode 100644 arch/x86/entry/entry_fred.c
> >  create mode 100644 arch/x86/include/asm/event-type.h
> >  create mode 100644 arch/x86/include/asm/fred.h
> >  create mode 100644 arch/x86/kernel/fred.c
> >