[PATCH 0/4] target/i386: add support for FRED

[PATCH 0/4] target/i386: add support for FRED

[Xin Li]

This patch set adds support for the Intel flexible return and event delivery
(FRED) architecture to allow Qemu to run KVM guests with FRED enabled.

The FRED architecture defines simple new transitions that change privilege
level (ring transitions). The FRED architecture was designed with the
following goals:
1) Improve overall performance and response time by replacing event delivery
   through the interrupt descriptor table (IDT event delivery) and event
   return by the IRET instruction with lower latency transitions.
2) Improve software robustness by ensuring that event delivery establishes
   the full supervisor context and that event return establishes the full
   user context.

Search for the latest FRED spec in most search engines with this search pattern:

  site:intel.com FRED (flexible return and event delivery) specification


Xin Li (4):
  target/i386: add support for FRED in CPUID enumeration
  target/i386: mark CR4.FRED not reserved
  target/i386: enumerate VMX nested-exception support
  target/i386: add live migration support for FRED

 target/i386/cpu.c     |  7 +++++-
 target/i386/cpu.h     | 40 +++++++++++++++++++++++++++++++-
 target/i386/kvm/kvm.c | 54 +++++++++++++++++++++++++++++++++++++++++++
 target/i386/machine.c | 10 ++++++++
 4 files changed, 109 insertions(+), 2 deletions(-)

-- 
2.34.1

[PATCH 1/4] target/i386: add support for FRED in CPUID enumeration

[Xin Li]

FRED, i.e., the Intel flexible return and event delivery architecture,
defines simple new transitions that change privilege level (ring
transitions).

In addition to these transitions, the FRED architecture defines a new
instruction (LKGS) for managing the state of the GS segment register.
The LKGS instruction can be used by 64-bit operating systems that do
not use the new FRED transitions.

The CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[17] enumerates FRED, and
the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[18] enumerates LKGS.

Add CPUID definitions for FRED/LKGS, and expose them to KVM guests only.

Because FRED relies on LKGS, add it to feature dependency map.

Tested-by: Shan Kang <shan.kang@intel.com>
Signed-off-by: Xin Li <xin3.li@intel.com>
---
 target/i386/cpu.c | 6 +++++-
 target/i386/cpu.h | 4 ++++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 00f913b638..3dba6b46d9 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -963,7 +963,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "avx-vnni", "avx512-bf16", NULL, "cmpccxadd",
             NULL, NULL, "fzrm", "fsrs",
             "fsrc", NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
+            NULL, "fred", "lkgs", NULL,
             NULL, "amx-fp16", NULL, "avx-ifma",
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
@@ -1549,6 +1549,10 @@ static FeatureDep feature_dependencies[] = {
         .from = { FEAT_VMX_SECONDARY_CTLS,  VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE },
         .to = { FEAT_7_0_ECX,               CPUID_7_0_ECX_WAITPKG },
     },
+    {
+        .from = { FEAT_7_1_EAX,             CPUID_7_1_EAX_LKGS },
+        .to = { FEAT_7_1_EAX,               CPUID_7_1_EAX_FRED },
+    },
 };
 
 typedef struct X86RegisterInfo32 {
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index a6000e93bd..064decbc85 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -932,6 +932,10 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,
 #define CPUID_7_1_EDX_AVX_NE_CONVERT    (1U << 5)
 /* PREFETCHIT0/1 Instructions */
 #define CPUID_7_1_EDX_PREFETCHITI       (1U << 14)
+/* Flexible return and event delivery (FRED) */
+#define CPUID_7_1_EAX_FRED              (1U << 17)
+/* Load into IA32_KERNEL_GS_BASE (LKGS) */
+#define CPUID_7_1_EAX_LKGS              (1U << 18)
 
 /* Do not exhibit MXCSR Configuration Dependent Timing (MCDT) behavior */
 #define CPUID_7_2_EDX_MCDT_NO           (1U << 5)
-- 
2.34.1

[PATCH 2/4] target/i386: mark CR4.FRED not reserved

[Xin Li]

The CR4.FRED bit, i.e., CR4[32], is no longer a reserved bit when FRED
is exposed to guests, otherwise it is still a reserved bit.

Tested-by: Shan Kang <shan.kang@intel.com>
Signed-off-by: Xin Li <xin3.li@intel.com>
---
 target/i386/cpu.h | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 064decbc85..924819a64c 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -262,6 +262,12 @@ typedef enum X86Seg {
 #define CR4_PKE_MASK   (1U << 22)
 #define CR4_PKS_MASK   (1U << 24)
 
+#ifdef TARGET_X86_64
+#define CR4_FRED_MASK   (1ULL << 32)
+#else
+#define CR4_FRED_MASK   0
+#endif
+
 #define CR4_RESERVED_MASK \
 (~(target_ulong)(CR4_VME_MASK | CR4_PVI_MASK | CR4_TSD_MASK \
                 | CR4_DE_MASK | CR4_PSE_MASK | CR4_PAE_MASK \
@@ -269,7 +275,8 @@ typedef enum X86Seg {
                 | CR4_OSFXSR_MASK | CR4_OSXMMEXCPT_MASK | CR4_UMIP_MASK \
                 | CR4_LA57_MASK \
                 | CR4_FSGSBASE_MASK | CR4_PCIDE_MASK | CR4_OSXSAVE_MASK \
-                | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK))
+                | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK \
+                | CR4_FRED_MASK))
 
 #define DR6_BD          (1 << 13)
 #define DR6_BS          (1 << 14)
@@ -2481,6 +2488,9 @@ static inline uint64_t cr4_reserved_bits(CPUX86State *env)
     if (!(env->features[FEAT_7_0_ECX] & CPUID_7_0_ECX_PKS)) {
         reserved_bits |= CR4_PKS_MASK;
     }
+    if (!(env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED)) {
+        reserved_bits |= CR4_FRED_MASK;
+    }
     return reserved_bits;
 }
 
-- 
2.34.1

[PATCH 3/4] target/i386: enumerate VMX nested-exception support

[Xin Li]

Allow VMX nested-exception support to be exposed in KVM guests, thus
nested KVM guests can enumerate it.

Tested-by: Shan Kang <shan.kang@intel.com>
Signed-off-by: Xin Li <xin3.li@intel.com>
---
 target/i386/cpu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 3dba6b46d9..ba579e1fb7 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1340,6 +1340,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         .feat_names = {
             [54] = "vmx-ins-outs",
             [55] = "vmx-true-ctls",
+            [58] = "vmx-nested-exception",
         },
         .msr = {
             .index = MSR_IA32_VMX_BASIC,
-- 
2.34.1

[PATCH 4/4] target/i386: add live migration support for FRED

[Xin Li]

FRED CPU states are managed in 10 FRED MSRs, in addtion to a few existing
CPU registers and MSRs, e.g., the CR4.FRED bit.

Add the 10 new FRED MSRs to x86 CPUArchState for live migration support.

Tested-by: Shan Kang <shan.kang@intel.com>
Signed-off-by: Xin Li <xin3.li@intel.com>
---
 target/i386/cpu.h     | 24 +++++++++++++++++++
 target/i386/kvm/kvm.c | 54 +++++++++++++++++++++++++++++++++++++++++++
 target/i386/machine.c | 10 ++++++++
 3 files changed, 88 insertions(+)

diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 924819a64c..a36a1a58c4 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -529,6 +529,20 @@ typedef enum X86Seg {
 #define MSR_IA32_XFD                    0x000001c4
 #define MSR_IA32_XFD_ERR                0x000001c5
 
+#define MSR_IA32_PL0_SSP                0x000006a4       /* Stack level 0 shadow stack pointer in ring 0 */
+
+/* FRED MSRs */
+#define MSR_IA32_FRED_RSP0              0x000001cc       /* Stack level 0 regular stack pointer */
+#define MSR_IA32_FRED_RSP1              0x000001cd       /* Stack level 1 regular stack pointer */
+#define MSR_IA32_FRED_RSP2              0x000001ce       /* Stack level 2 regular stack pointer */
+#define MSR_IA32_FRED_RSP3              0x000001cf       /* Stack level 3 regular stack pointer */
+#define MSR_IA32_FRED_STKLVLS           0x000001d0       /* FRED exception stack levels */
+#define MSR_IA32_FRED_SSP0              MSR_IA32_PL0_SSP /* Stack level 0 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP1              0x000001d1       /* Stack level 1 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP2              0x000001d2       /* Stack level 2 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_SSP3              0x000001d3       /* Stack level 3 shadow stack pointer in ring 0 */
+#define MSR_IA32_FRED_CONFIG            0x000001d4       /* FRED Entrypoint and interrupt stack level */
+
 #define MSR_IA32_BNDCFGS                0x00000d90
 #define MSR_IA32_XSS                    0x00000da0
 #define MSR_IA32_UMWAIT_CONTROL         0xe1
@@ -1680,6 +1694,16 @@ typedef struct CPUArchState {
     target_ulong cstar;
     target_ulong fmask;
     target_ulong kernelgsbase;
+    target_ulong fred_rsp0;
+    target_ulong fred_rsp1;
+    target_ulong fred_rsp2;
+    target_ulong fred_rsp3;
+    target_ulong fred_stklvls;
+    target_ulong fred_ssp0;
+    target_ulong fred_ssp1;
+    target_ulong fred_ssp2;
+    target_ulong fred_ssp3;
+    target_ulong fred_config;
 #endif
 
     uint64_t tsc_adjust;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 639a242ad8..4b241c82d8 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -3401,6 +3401,18 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
         kvm_msr_entry_add(cpu, MSR_KERNELGSBASE, env->kernelgsbase);
         kvm_msr_entry_add(cpu, MSR_FMASK, env->fmask);
         kvm_msr_entry_add(cpu, MSR_LSTAR, env->lstar);
+        if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED) {
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP0, env->fred_rsp0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP1, env->fred_rsp1);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP2, env->fred_rsp2);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP3, env->fred_rsp3);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_STKLVLS, env->fred_stklvls);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP0, env->fred_ssp0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP1, env->fred_ssp1);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP2, env->fred_ssp2);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP3, env->fred_ssp3);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_CONFIG, env->fred_config);
+        }
     }
 #endif
 
@@ -3901,6 +3913,18 @@ static int kvm_get_msrs(X86CPU *cpu)
         kvm_msr_entry_add(cpu, MSR_KERNELGSBASE, 0);
         kvm_msr_entry_add(cpu, MSR_FMASK, 0);
         kvm_msr_entry_add(cpu, MSR_LSTAR, 0);
+        if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED) {
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP0, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP1, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP2, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP3, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_STKLVLS, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP0, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP1, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP2, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP3, 0);
+            kvm_msr_entry_add(cpu, MSR_IA32_FRED_CONFIG, 0);
+        }
     }
 #endif
     kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, 0);
@@ -4123,6 +4147,36 @@ static int kvm_get_msrs(X86CPU *cpu)
         case MSR_LSTAR:
             env->lstar = msrs[i].data;
             break;
+        case MSR_IA32_FRED_RSP0:
+            env->fred_rsp0 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_RSP1:
+            env->fred_rsp1 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_RSP2:
+            env->fred_rsp2 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_RSP3:
+            env->fred_rsp3 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_STKLVLS:
+            env->fred_stklvls = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP0:
+            env->fred_ssp0 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP1:
+            env->fred_ssp1 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP2:
+            env->fred_ssp2 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_SSP3:
+            env->fred_ssp3 = msrs[i].data;
+            break;
+        case MSR_IA32_FRED_CONFIG:
+            env->fred_config = msrs[i].data;
+            break;
 #endif
         case MSR_IA32_TSC:
             env->tsc = msrs[i].data;
diff --git a/target/i386/machine.c b/target/i386/machine.c
index c7ac8084b2..5c722a49c5 100644
--- a/target/i386/machine.c
+++ b/target/i386/machine.c
@@ -1652,6 +1652,16 @@ const VMStateDescription vmstate_x86_cpu = {
         VMSTATE_UINT64(env.cstar, X86CPU),
         VMSTATE_UINT64(env.fmask, X86CPU),
         VMSTATE_UINT64(env.kernelgsbase, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp0, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp1, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp2, X86CPU),
+        VMSTATE_UINT64(env.fred_rsp3, X86CPU),
+        VMSTATE_UINT64(env.fred_stklvls, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp0, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp1, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp2, X86CPU),
+        VMSTATE_UINT64(env.fred_ssp3, X86CPU),
+        VMSTATE_UINT64(env.fred_config, X86CPU),
 #endif
         VMSTATE_UINT32(env.smbase, X86CPU),
 
-- 
2.34.1

Re: [PATCH 4/4] target/i386: add live migration support for FRED

[Yang, Weijiang]

On 9/1/2023 1:30 PM, Li, Xin3 wrote:
> FRED CPU states are managed in 10 FRED MSRs, in addtion to a few existing
> CPU registers and MSRs, e.g., the CR4.FRED bit.
>
> Add the 10 new FRED MSRs to x86 CPUArchState for live migration support.
>
> Tested-by: Shan Kang <shan.kang@intel.com>
> Signed-off-by: Xin Li <xin3.li@intel.com>
> ---
>   target/i386/cpu.h     | 24 +++++++++++++++++++
>   target/i386/kvm/kvm.c | 54 +++++++++++++++++++++++++++++++++++++++++++
>   target/i386/machine.c | 10 ++++++++
>   3 files changed, 88 insertions(+)
>
> diff --git a/target/i386/cpu.h b/target/i386/cpu.h
> index 924819a64c..a36a1a58c4 100644
> --- a/target/i386/cpu.h
> +++ b/target/i386/cpu.h
> @@ -529,6 +529,20 @@ typedef enum X86Seg {
>   #define MSR_IA32_XFD                    0x000001c4
>   #define MSR_IA32_XFD_ERR                0x000001c5
>   
> +#define MSR_IA32_PL0_SSP                0x000006a4       /* Stack level 0 shadow stack pointer in ring 0 */
> +
> +/* FRED MSRs */
> +#define MSR_IA32_FRED_RSP0              0x000001cc       /* Stack level 0 regular stack pointer */
> +#define MSR_IA32_FRED_RSP1              0x000001cd       /* Stack level 1 regular stack pointer */
> +#define MSR_IA32_FRED_RSP2              0x000001ce       /* Stack level 2 regular stack pointer */
> +#define MSR_IA32_FRED_RSP3              0x000001cf       /* Stack level 3 regular stack pointer */
> +#define MSR_IA32_FRED_STKLVLS           0x000001d0       /* FRED exception stack levels */
> +#define MSR_IA32_FRED_SSP0              MSR_IA32_PL0_SSP /* Stack level 0 shadow stack pointer in ring 0 */
> +#define MSR_IA32_FRED_SSP1              0x000001d1       /* Stack level 1 shadow stack pointer in ring 0 */
> +#define MSR_IA32_FRED_SSP2              0x000001d2       /* Stack level 2 shadow stack pointer in ring 0 */
> +#define MSR_IA32_FRED_SSP3              0x000001d3       /* Stack level 3 shadow stack pointer in ring 0 */
> +#define MSR_IA32_FRED_CONFIG            0x000001d4       /* FRED Entrypoint and interrupt stack level */
> +
>   #define MSR_IA32_BNDCFGS                0x00000d90
>   #define MSR_IA32_XSS                    0x00000da0
>   #define MSR_IA32_UMWAIT_CONTROL         0xe1
> @@ -1680,6 +1694,16 @@ typedef struct CPUArchState {
>       target_ulong cstar;
>       target_ulong fmask;
>       target_ulong kernelgsbase;
> +    target_ulong fred_rsp0;
> +    target_ulong fred_rsp1;
> +    target_ulong fred_rsp2;
> +    target_ulong fred_rsp3;
> +    target_ulong fred_stklvls;
> +    target_ulong fred_ssp0;
> +    target_ulong fred_ssp1;
> +    target_ulong fred_ssp2;
> +    target_ulong fred_ssp3;
> +    target_ulong fred_config;
>   #endif
>   
>       uint64_t tsc_adjust;
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index 639a242ad8..4b241c82d8 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -3401,6 +3401,18 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
>           kvm_msr_entry_add(cpu, MSR_KERNELGSBASE, env->kernelgsbase);
>           kvm_msr_entry_add(cpu, MSR_FMASK, env->fmask);
>           kvm_msr_entry_add(cpu, MSR_LSTAR, env->lstar);
> +        if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED) {
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP0, env->fred_rsp0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP1, env->fred_rsp1);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP2, env->fred_rsp2);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP3, env->fred_rsp3);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_STKLVLS, env->fred_stklvls);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP0, env->fred_ssp0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP1, env->fred_ssp1);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP2, env->fred_ssp2);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP3, env->fred_ssp3);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_CONFIG, env->fred_config);
> +        }
>       }
>   #endif
>   
> @@ -3901,6 +3913,18 @@ static int kvm_get_msrs(X86CPU *cpu)
>           kvm_msr_entry_add(cpu, MSR_KERNELGSBASE, 0);
>           kvm_msr_entry_add(cpu, MSR_FMASK, 0);
>           kvm_msr_entry_add(cpu, MSR_LSTAR, 0);
> +        if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_FRED) {
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP0, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP1, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP2, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_RSP3, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_STKLVLS, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP0, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP1, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP2, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_SSP3, 0);
> +            kvm_msr_entry_add(cpu, MSR_IA32_FRED_CONFIG, 0);
> +        }
>       }
>   #endif
>       kvm_msr_entry_add(cpu, MSR_KVM_SYSTEM_TIME, 0);
> @@ -4123,6 +4147,36 @@ static int kvm_get_msrs(X86CPU *cpu)
>           case MSR_LSTAR:
>               env->lstar = msrs[i].data;
>               break;
> +        case MSR_IA32_FRED_RSP0:
> +            env->fred_rsp0 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_RSP1:
> +            env->fred_rsp1 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_RSP2:
> +            env->fred_rsp2 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_RSP3:
> +            env->fred_rsp3 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_STKLVLS:
> +            env->fred_stklvls = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_SSP0:
> +            env->fred_ssp0 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_SSP1:
> +            env->fred_ssp1 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_SSP2:
> +            env->fred_ssp2 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_SSP3:
> +            env->fred_ssp3 = msrs[i].data;
> +            break;
> +        case MSR_IA32_FRED_CONFIG:
> +            env->fred_config = msrs[i].data;
> +            break;
>   #endif
>           case MSR_IA32_TSC:
>               env->tsc = msrs[i].data;
> diff --git a/target/i386/machine.c b/target/i386/machine.c
> index c7ac8084b2..5c722a49c5 100644
> --- a/target/i386/machine.c
> +++ b/target/i386/machine.c
> @@ -1652,6 +1652,16 @@ const VMStateDescription vmstate_x86_cpu = {
>           VMSTATE_UINT64(env.cstar, X86CPU),
>           VMSTATE_UINT64(env.fmask, X86CPU),
>           VMSTATE_UINT64(env.kernelgsbase, X86CPU),
> +        VMSTATE_UINT64(env.fred_rsp0, X86CPU),
> +        VMSTATE_UINT64(env.fred_rsp1, X86CPU),
> +        VMSTATE_UINT64(env.fred_rsp2, X86CPU),
> +        VMSTATE_UINT64(env.fred_rsp3, X86CPU),
> +        VMSTATE_UINT64(env.fred_stklvls, X86CPU),
> +        VMSTATE_UINT64(env.fred_ssp0, X86CPU),
> +        VMSTATE_UINT64(env.fred_ssp1, X86CPU),
> +        VMSTATE_UINT64(env.fred_ssp2, X86CPU),
> +        VMSTATE_UINT64(env.fred_ssp3, X86CPU),
> +        VMSTATE_UINT64(env.fred_config, X86CPU),
>   #endif
>           VMSTATE_UINT32(env.smbase, X86CPU),
>   

IMO, it's better to split this patch into two, one is for FRED MSR access, the other is
for LM support.

And move the FRED MSRs from basic x86_cpu part to .subsections part because FRED
doesn't belong to basic CPU registers.  In the .subsections part, you may define a struct
to hold FRED MSRs then add  .needed callback helper for QEMU to query whether FRED
MSRs are necessary for live migration.

See other features, e.g., vmstate_msr_intel_sgx

RE: [PATCH 4/4] target/i386: add live migration support for FRED

[Li, Xin3]

> > FRED CPU states are managed in 10 FRED MSRs, in addtion to a few
> > existing CPU registers and MSRs, e.g., the CR4.FRED bit.
> >
> > Add the 10 new FRED MSRs to x86 CPUArchState for live migration support.
> >
> 
> IMO, it's better to split this patch into two, one is for FRED MSR access, the other
> is for LM support.
> 
> And move the FRED MSRs from basic x86_cpu part to .subsections part because
> FRED doesn't belong to basic CPU registers.  In the .subsections part, you may
> define a struct to hold FRED MSRs then add  .needed callback helper for QEMU to
> query whether FRED MSRs are necessary for live migration.

Good point!

> 
> See other features, e.g., vmstate_msr_intel_sgx

Re: [PATCH 3/4] target/i386: enumerate VMX nested-exception support

[Paolo Bonzini]

On 9/1/23 07:30, Xin Li wrote:
> Allow VMX nested-exception support to be exposed in KVM guests, thus
> nested KVM guests can enumerate it.
> 
> Tested-by: Shan Kang <shan.kang@intel.com>
> Signed-off-by: Xin Li <xin3.li@intel.com>
> ---
>   target/i386/cpu.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 3dba6b46d9..ba579e1fb7 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -1340,6 +1340,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
>           .feat_names = {
>               [54] = "vmx-ins-outs",
>               [55] = "vmx-true-ctls",
> +            [58] = "vmx-nested-exception",
>           },
>           .msr = {
>               .index = MSR_IA32_VMX_BASIC,

Please also add it to scripts/kvm/vmxcap, and rebase on top of the 
recent introduction of MSR_VMX_BASIC_ANY_ERRCODE.

Paolo