[PATCH 0/3] Add generic MMIO instruction deconding to be used in SEV and TDX

[PATCH 0/3] Add generic MMIO instruction deconding to be used in SEV and TDX

[Kirill A. Shutemov]

Both AMD SEV and Intel TDX has to decode MMIO instruction to be able to
handle MMIO.

Extract insn_decode_mmio() from SEV code. TDX will also use this helper.

Kirill A. Shutemov (3):
  x86/insn-eval: Introduce insn_get_modrm_reg_ptr()
  x86/insn-eval: Introduce insn_decode_mmio()
  x86/sev-es: Use insn_decode_mmio() for MMIO implementation

 arch/x86/include/asm/insn-eval.h |  13 +++
 arch/x86/kernel/sev.c            | 171 ++++++++-----------------------
 arch/x86/lib/insn-eval.c         | 102 ++++++++++++++++++
 3 files changed, 155 insertions(+), 131 deletions(-)

-- 
2.32.0

[PATCH 1/3] x86/insn-eval: Introduce insn_get_modrm_reg_ptr()

[Kirill A. Shutemov]

The helper returns a pointer to the register indicated by
ModRM byte.

It's going to replace vc_insn_get_reg() in the SEV MMIO
implementation. TDX MMIO implementation will also use it.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reviewed-by: Andi Kleen <ak@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
---
 arch/x86/include/asm/insn-eval.h |  1 +
 arch/x86/lib/insn-eval.c         | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 91d7182ad2d6..041f399153b9 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -19,6 +19,7 @@ bool insn_has_rep_prefix(struct insn *insn);
 void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
 int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
 int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs);
+void *insn_get_modrm_reg_ptr(struct insn *insn, struct pt_regs *regs);
 unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
 int insn_get_code_seg_params(struct pt_regs *regs);
 int insn_fetch_from_user(struct pt_regs *regs,
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index a1d24fdc07cf..fbaa3fa24bde 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -850,6 +850,26 @@ int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs)
 	return get_reg_offset(insn, regs, REG_TYPE_REG);
 }
 
+/**
+ * insn_get_modrm_reg_ptr() - Obtain register pointer based on ModRM byte
+ * @insn:	Instruction containing the ModRM byte
+ * @regs:	Register values as seen when entering kernel mode
+ *
+ * Returns:
+ *
+ * The register indicated by the reg part of the ModRM byte.
+ * The register is obtained as a pointer within pt_regs.
+ */
+void *insn_get_modrm_reg_ptr(struct insn *insn, struct pt_regs *regs)
+{
+	int offset;
+
+	offset = insn_get_modrm_reg_off(insn, regs);
+	if (offset < 0)
+		return NULL;
+	return (void *)regs + offset;
+}
+
 /**
  * get_seg_base_limit() - obtain base address and limit of a segment
  * @insn:	Instruction. Must be valid.
-- 
2.32.0

Re: [PATCH 1/3] x86/insn-eval: Introduce insn_get_modrm_reg_ptr()

[Peter Zijlstra]

On Mon, Oct 18, 2021 at 06:33:31PM +0300, Kirill A. Shutemov wrote:
> +/**
> + * insn_get_modrm_reg_ptr() - Obtain register pointer based on ModRM byte
> + * @insn:	Instruction containing the ModRM byte
> + * @regs:	Register values as seen when entering kernel mode
> + *
> + * Returns:
> + *
> + * The register indicated by the reg part of the ModRM byte.
> + * The register is obtained as a pointer within pt_regs.
> + */
> +void *insn_get_modrm_reg_ptr(struct insn *insn, struct pt_regs *regs)

Doesn't that return type want to be 'unsigned long *'?

> +{
> +	int offset;
> +
> +	offset = insn_get_modrm_reg_off(insn, regs);
> +	if (offset < 0)
> +		return NULL;
> +	return (void *)regs + offset;
> +}
> +
>  /**
>   * get_seg_base_limit() - obtain base address and limit of a segment
>   * @insn:	Instruction. Must be valid.
> -- 
> 2.32.0
> 

Re: [PATCH 1/3] x86/insn-eval: Introduce insn_get_modrm_reg_ptr()

[Kirill A. Shutemov]

On Mon, Oct 18, 2021 at 05:47:17PM +0200, Peter Zijlstra wrote:
> On Mon, Oct 18, 2021 at 06:33:31PM +0300, Kirill A. Shutemov wrote:
> > +/**
> > + * insn_get_modrm_reg_ptr() - Obtain register pointer based on ModRM byte
> > + * @insn:	Instruction containing the ModRM byte
> > + * @regs:	Register values as seen when entering kernel mode
> > + *
> > + * Returns:
> > + *
> > + * The register indicated by the reg part of the ModRM byte.
> > + * The register is obtained as a pointer within pt_regs.
> > + */
> > +void *insn_get_modrm_reg_ptr(struct insn *insn, struct pt_regs *regs)
> 
> Doesn't that return type want to be 'unsigned long *'?

Right, will fix.

-- 
 Kirill A. Shutemov

[PATCH 2/3] x86/insn-eval: Introduce insn_decode_mmio()

[Kirill A. Shutemov]

In preparation for sharing MMIO instruction decode between SEV-ES and
TDX, factor out the common decode into a new insn_decode_mmio() helper.

For regular virtual machine, MMIO is handled by the VMM and KVM
emulates instructions that caused MMIO. But, this model doesn't work
for a secure VMs (like SEV or TDX) as VMM doesn't have access to the
guest memory and register state. So, for TDX or SEV VMM needs
assistance in handling MMIO. It induces exception in the guest. Guest
has to decode the instruction and handle it on its own.

The code is based on the current SEV MMIO implementation.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reviewed-by: Andi Kleen <ak@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
---
 arch/x86/include/asm/insn-eval.h | 12 +++++
 arch/x86/lib/insn-eval.c         | 82 ++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+)

diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 041f399153b9..4a4ca7e7be66 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -29,4 +29,16 @@ int insn_fetch_from_user_inatomic(struct pt_regs *regs,
 bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs,
 			   unsigned char buf[MAX_INSN_SIZE], int buf_size);
 
+enum mmio_type {
+	MMIO_DECODE_FAILED,
+	MMIO_WRITE,
+	MMIO_WRITE_IMM,
+	MMIO_READ,
+	MMIO_READ_ZERO_EXTEND,
+	MMIO_READ_SIGN_EXTEND,
+	MMIO_MOVS,
+};
+
+enum mmio_type insn_decode_mmio(struct insn *insn, int *bytes);
+
 #endif /* _ASM_X86_INSN_EVAL_H */
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index fbaa3fa24bde..2ab29d8d6731 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -1559,3 +1559,85 @@ bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs,
 
 	return true;
 }
+
+/**
+ * insn_decode_mmio() - Decode a MMIO instruction
+ * @insn:	Structure to store decoded instruction
+ * @bytes:	Returns size of memory operand
+ *
+ * Decodes instruction that used for Memory-mapped I/O.
+ *
+ * Returns:
+ *
+ * Type of the instruction. Size of the memory operand is stored in
+ * @bytes. If decode failed, MMIO_DECODE_FAILED returned.
+ */
+enum mmio_type insn_decode_mmio(struct insn *insn, int *bytes)
+{
+	int type = MMIO_DECODE_FAILED;
+
+	*bytes = 0;
+
+	insn_get_opcode(insn);
+	switch (insn->opcode.bytes[0]) {
+	case 0x88: /* MOV m8,r8 */
+		*bytes = 1;
+		fallthrough;
+	case 0x89: /* MOV m16/m32/m64, r16/m32/m64 */
+		if (!*bytes)
+			*bytes = insn->opnd_bytes;
+		type = MMIO_WRITE;
+		break;
+
+	case 0xc6: /* MOV m8, imm8 */
+		*bytes = 1;
+		fallthrough;
+	case 0xc7: /* MOV m16/m32/m64, imm16/imm32/imm64 */
+		if (!*bytes)
+			*bytes = insn->opnd_bytes;
+		type = MMIO_WRITE_IMM;
+		break;
+
+	case 0x8a: /* MOV r8, m8 */
+		*bytes = 1;
+		fallthrough;
+	case 0x8b: /* MOV r16/r32/r64, m16/m32/m64 */
+		if (!*bytes)
+			*bytes = insn->opnd_bytes;
+		type = MMIO_READ;
+		break;
+
+	case 0xa4: /* MOVS m8, m8 */
+		*bytes = 1;
+		fallthrough;
+	case 0xa5: /* MOVS m16/m32/m64, m16/m32/m64 */
+		if (!*bytes)
+			*bytes = insn->opnd_bytes;
+		type = MMIO_MOVS;
+		break;
+
+	case 0x0f: /* Two-byte instruction */
+		switch (insn->opcode.bytes[1]) {
+		case 0xb6: /* MOVZX r16/r32/r64, m8 */
+			*bytes = 1;
+			fallthrough;
+		case 0xb7: /* MOVZX r32/r64, m16 */
+			if (!*bytes)
+				*bytes = 2;
+			type = MMIO_READ_ZERO_EXTEND;
+			break;
+
+		case 0xbe: /* MOVSX r16/r32/r64, m8 */
+			*bytes = 1;
+			fallthrough;
+		case 0xbf: /* MOVSX r32/r64, m16 */
+			if (!*bytes)
+				*bytes = 2;
+			type = MMIO_READ_SIGN_EXTEND;
+			break;
+		}
+		break;
+	}
+
+	return type;
+}
-- 
2.32.0

Re: [PATCH 2/3] x86/insn-eval: Introduce insn_decode_mmio()

[Peter Zijlstra]

On Mon, Oct 18, 2021 at 06:33:32PM +0300, Kirill A. Shutemov wrote:

> diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
> index fbaa3fa24bde..2ab29d8d6731 100644
> --- a/arch/x86/lib/insn-eval.c
> +++ b/arch/x86/lib/insn-eval.c
> @@ -1559,3 +1559,85 @@ bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs,
>  
>  	return true;
>  }
> +
> +/**
> + * insn_decode_mmio() - Decode a MMIO instruction
> + * @insn:	Structure to store decoded instruction
> + * @bytes:	Returns size of memory operand
> + *
> + * Decodes instruction that used for Memory-mapped I/O.
> + *
> + * Returns:
> + *
> + * Type of the instruction. Size of the memory operand is stored in
> + * @bytes. If decode failed, MMIO_DECODE_FAILED returned.
> + */
> +enum mmio_type insn_decode_mmio(struct insn *insn, int *bytes)
> +{
> +	int type = MMIO_DECODE_FAILED;
> +
> +	*bytes = 0;
> +
> +	insn_get_opcode(insn);

insn_get_opcode() can fail. Either you assume it's already called and
don't call it, or you can't assume anything and get to do error
handling.

> +	switch (insn->opcode.bytes[0]) {
...
> +	}
> +
> +	return type;
> +}
> -- 
> 2.32.0
> 

Re: [PATCH 2/3] x86/insn-eval: Introduce insn_decode_mmio()

[Kirill A. Shutemov]

On Mon, Oct 18, 2021 at 05:53:49PM +0200, Peter Zijlstra wrote:
> On Mon, Oct 18, 2021 at 06:33:32PM +0300, Kirill A. Shutemov wrote:
> 
> > diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
> > index fbaa3fa24bde..2ab29d8d6731 100644
> > --- a/arch/x86/lib/insn-eval.c
> > +++ b/arch/x86/lib/insn-eval.c
> > @@ -1559,3 +1559,85 @@ bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs,
> >  
> >  	return true;
> >  }
> > +
> > +/**
> > + * insn_decode_mmio() - Decode a MMIO instruction
> > + * @insn:	Structure to store decoded instruction
> > + * @bytes:	Returns size of memory operand
> > + *
> > + * Decodes instruction that used for Memory-mapped I/O.
> > + *
> > + * Returns:
> > + *
> > + * Type of the instruction. Size of the memory operand is stored in
> > + * @bytes. If decode failed, MMIO_DECODE_FAILED returned.
> > + */
> > +enum mmio_type insn_decode_mmio(struct insn *insn, int *bytes)
> > +{
> > +	int type = MMIO_DECODE_FAILED;
> > +
> > +	*bytes = 0;
> > +
> > +	insn_get_opcode(insn);
> 
> insn_get_opcode() can fail. Either you assume it's already called and
> don't call it, or you can't assume anything and get to do error
> handling.

Fair enough. I will return MMIO_DECODE_FAILED if insn_get_opcode() fails.

BTW, looks like is_string_insn() suffers from the same issue. Not sure how
to fix it though.

-- 
 Kirill A. Shutemov

Re: [PATCH 2/3] x86/insn-eval: Introduce insn_decode_mmio()

[Peter Zijlstra]

On Mon, Oct 18, 2021 at 07:49:41PM +0300, Kirill A. Shutemov wrote:
> On Mon, Oct 18, 2021 at 05:53:49PM +0200, Peter Zijlstra wrote:
> > On Mon, Oct 18, 2021 at 06:33:32PM +0300, Kirill A. Shutemov wrote:
> > 
> > > diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
> > > index fbaa3fa24bde..2ab29d8d6731 100644
> > > --- a/arch/x86/lib/insn-eval.c
> > > +++ b/arch/x86/lib/insn-eval.c
> > > @@ -1559,3 +1559,85 @@ bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs,
> > >  
> > >  	return true;
> > >  }
> > > +
> > > +/**
> > > + * insn_decode_mmio() - Decode a MMIO instruction
> > > + * @insn:	Structure to store decoded instruction
> > > + * @bytes:	Returns size of memory operand
> > > + *
> > > + * Decodes instruction that used for Memory-mapped I/O.
> > > + *
> > > + * Returns:
> > > + *
> > > + * Type of the instruction. Size of the memory operand is stored in
> > > + * @bytes. If decode failed, MMIO_DECODE_FAILED returned.
> > > + */
> > > +enum mmio_type insn_decode_mmio(struct insn *insn, int *bytes)
> > > +{
> > > +	int type = MMIO_DECODE_FAILED;
> > > +
> > > +	*bytes = 0;
> > > +
> > > +	insn_get_opcode(insn);
> > 
> > insn_get_opcode() can fail. Either you assume it's already called and
> > don't call it, or you can't assume anything and get to do error
> > handling.
> 
> Fair enough. I will return MMIO_DECODE_FAILED if insn_get_opcode() fails.
> 
> BTW, looks like is_string_insn() suffers from the same issue. Not sure how
> to fix it though.

AFAICT all callers of insn_get_addr_ref() (which is what
is_string_insn() seems to be part of) do a insn_decode_*() call with
error checking before.

So it looks like that insn_get_opcode() in there is superfluous.
probably same for insn_has_rep_prefix() / get_seg_reg_override_idx() /
get_eff_addr_modrm_*(). That all wants cleaning up.

The esaiest way is probably to push those things up the callchains into
the !static function and have it fail early there.

Re: [PATCH 2/3] x86/insn-eval: Introduce insn_decode_mmio()

[Kirill A. Shutemov]

On Tue, Oct 19, 2021 at 10:34:45AM +0200, Peter Zijlstra wrote:
> On Mon, Oct 18, 2021 at 07:49:41PM +0300, Kirill A. Shutemov wrote:
> > On Mon, Oct 18, 2021 at 05:53:49PM +0200, Peter Zijlstra wrote:
> > > On Mon, Oct 18, 2021 at 06:33:32PM +0300, Kirill A. Shutemov wrote:
> > > 
> > > > diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
> > > > index fbaa3fa24bde..2ab29d8d6731 100644
> > > > --- a/arch/x86/lib/insn-eval.c
> > > > +++ b/arch/x86/lib/insn-eval.c
> > > > @@ -1559,3 +1559,85 @@ bool insn_decode_from_regs(struct insn *insn, struct pt_regs *regs,
> > > >  
> > > >  	return true;
> > > >  }
> > > > +
> > > > +/**
> > > > + * insn_decode_mmio() - Decode a MMIO instruction
> > > > + * @insn:	Structure to store decoded instruction
> > > > + * @bytes:	Returns size of memory operand
> > > > + *
> > > > + * Decodes instruction that used for Memory-mapped I/O.
> > > > + *
> > > > + * Returns:
> > > > + *
> > > > + * Type of the instruction. Size of the memory operand is stored in
> > > > + * @bytes. If decode failed, MMIO_DECODE_FAILED returned.
> > > > + */
> > > > +enum mmio_type insn_decode_mmio(struct insn *insn, int *bytes)
> > > > +{
> > > > +	int type = MMIO_DECODE_FAILED;
> > > > +
> > > > +	*bytes = 0;
> > > > +
> > > > +	insn_get_opcode(insn);
> > > 
> > > insn_get_opcode() can fail. Either you assume it's already called and
> > > don't call it, or you can't assume anything and get to do error
> > > handling.
> > 
> > Fair enough. I will return MMIO_DECODE_FAILED if insn_get_opcode() fails.
> > 
> > BTW, looks like is_string_insn() suffers from the same issue. Not sure how
> > to fix it though.
> 
> AFAICT all callers of insn_get_addr_ref() (which is what
> is_string_insn() seems to be part of) do a insn_decode_*() call with
> error checking before.
> 
> So it looks like that insn_get_opcode() in there is superfluous.
> probably same for insn_has_rep_prefix() / get_seg_reg_override_idx() /
> get_eff_addr_modrm_*(). That all wants cleaning up.
> 
> The esaiest way is probably to push those things up the callchains into
> the !static function and have it fail early there.

Looks like insn_get_addr_ref() is the only non-static callsite:

	is_string_insn
	  check_seg_overrides
	    resolve_seg_reg
	      get_seg_base_limit
		get_addr_ref_XX
		  insn_get_addr_ref
	  resolve_default_seg
	    resolve_seg_reg (as above)

What about this:

From 576a390de4759f591709c9961b5d7add02c814f0 Mon Sep 17 00:00:00 2001
From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Date: Sat, 23 Oct 2021 19:26:07 +0300
Subject: [PATCH] x86/insn-eval: Handle insn_get_opcode() failure

is_string_insn() calls insn_get_opcode() that can fail, but does not
handle the failure.

is_string_insn() interface does not allow to communicate an error to the
caller.

Push insn_get_opcode() to the only non-static user of is_string_insn()
and fail it early if insn_get_opcode() fails.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/lib/insn-eval.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 1714359f48bd..ccbc7b253d98 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -37,8 +37,6 @@ enum reg_type {
  */
 static bool is_string_insn(struct insn *insn)
 {
-	insn_get_opcode(insn);
-
 	/* All string instructions have a 1-byte opcode. */
 	if (insn->opcode.nbytes != 1)
 		return false;
@@ -1425,6 +1423,9 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs)
 	if (!insn || !regs)
 		return (void __user *)-1L;
 
+	if (insn_get_opcode(insn))
+		return (void __user *)-1L;
+
 	switch (insn->addr_bytes) {
 	case 2:
 		return get_addr_ref_16(insn, regs);
-- 
 Kirill A. Shutemov

[PATCH 3/3] x86/sev-es: Use insn_decode_mmio() for MMIO implementation

[Kirill A. Shutemov]

Switch SEV implementation to insn_decode_mmio(). The helper is going
to be used by TDX too.

No functional changes. It is only build-tested.

Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Reviewed-by: Andi Kleen <ak@linux.intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
---
 arch/x86/kernel/sev.c | 171 ++++++++++--------------------------------
 1 file changed, 40 insertions(+), 131 deletions(-)

diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c
index a6895e440bc3..8fea7ea67c67 100644
--- a/arch/x86/kernel/sev.c
+++ b/arch/x86/kernel/sev.c
@@ -807,22 +807,6 @@ static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
 	do_early_exception(ctxt->regs, trapnr);
 }
 
-static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)
-{
-	long *reg_array;
-	int offset;
-
-	reg_array = (long *)ctxt->regs;
-	offset    = insn_get_modrm_reg_off(&ctxt->insn, ctxt->regs);
-
-	if (offset < 0)
-		return NULL;
-
-	offset /= sizeof(long);
-
-	return reg_array + offset;
-}
-
 static long *vc_insn_get_rm(struct es_em_ctxt *ctxt)
 {
 	long *reg_array;
@@ -870,76 +854,6 @@ static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
 	return sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, exit_info_1, exit_info_2);
 }
 
-static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,
-						 struct es_em_ctxt *ctxt)
-{
-	struct insn *insn = &ctxt->insn;
-	unsigned int bytes = 0;
-	enum es_result ret;
-	int sign_byte;
-	long *reg_data;
-
-	switch (insn->opcode.bytes[1]) {
-		/* MMIO Read w/ zero-extension */
-	case 0xb6:
-		bytes = 1;
-		fallthrough;
-	case 0xb7:
-		if (!bytes)
-			bytes = 2;
-
-		ret = vc_do_mmio(ghcb, ctxt, bytes, true);
-		if (ret)
-			break;
-
-		/* Zero extend based on operand size */
-		reg_data = vc_insn_get_reg(ctxt);
-		if (!reg_data)
-			return ES_DECODE_FAILED;
-
-		memset(reg_data, 0, insn->opnd_bytes);
-
-		memcpy(reg_data, ghcb->shared_buffer, bytes);
-		break;
-
-		/* MMIO Read w/ sign-extension */
-	case 0xbe:
-		bytes = 1;
-		fallthrough;
-	case 0xbf:
-		if (!bytes)
-			bytes = 2;
-
-		ret = vc_do_mmio(ghcb, ctxt, bytes, true);
-		if (ret)
-			break;
-
-		/* Sign extend based on operand size */
-		reg_data = vc_insn_get_reg(ctxt);
-		if (!reg_data)
-			return ES_DECODE_FAILED;
-
-		if (bytes == 1) {
-			u8 *val = (u8 *)ghcb->shared_buffer;
-
-			sign_byte = (*val & 0x80) ? 0xff : 0x00;
-		} else {
-			u16 *val = (u16 *)ghcb->shared_buffer;
-
-			sign_byte = (*val & 0x8000) ? 0xff : 0x00;
-		}
-		memset(reg_data, sign_byte, insn->opnd_bytes);
-
-		memcpy(reg_data, ghcb->shared_buffer, bytes);
-		break;
-
-	default:
-		ret = ES_UNSUPPORTED;
-	}
-
-	return ret;
-}
-
 /*
  * The MOVS instruction has two memory operands, which raises the
  * problem that it is not known whether the access to the source or the
@@ -1007,83 +921,78 @@ static enum es_result vc_handle_mmio_movs(struct es_em_ctxt *ctxt,
 		return ES_RETRY;
 }
 
-static enum es_result vc_handle_mmio(struct ghcb *ghcb,
-				     struct es_em_ctxt *ctxt)
+static enum es_result vc_handle_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 {
 	struct insn *insn = &ctxt->insn;
 	unsigned int bytes = 0;
+	enum mmio_type mmio;
 	enum es_result ret;
+	u8 sign_byte;
 	long *reg_data;
 
-	switch (insn->opcode.bytes[0]) {
-	/* MMIO Write */
-	case 0x88:
-		bytes = 1;
-		fallthrough;
-	case 0x89:
-		if (!bytes)
-			bytes = insn->opnd_bytes;
+	mmio = insn_decode_mmio(insn, &bytes);
+	if (mmio == MMIO_DECODE_FAILED)
+		return ES_DECODE_FAILED;
 
-		reg_data = vc_insn_get_reg(ctxt);
+	if (mmio != MMIO_WRITE_IMM && mmio != MMIO_MOVS) {
+		reg_data = insn_get_modrm_reg_ptr(insn, ctxt->regs);
 		if (!reg_data)
 			return ES_DECODE_FAILED;
+	}
 
+	switch (mmio) {
+	case MMIO_WRITE:
 		memcpy(ghcb->shared_buffer, reg_data, bytes);
-
 		ret = vc_do_mmio(ghcb, ctxt, bytes, false);
 		break;
-
-	case 0xc6:
-		bytes = 1;
-		fallthrough;
-	case 0xc7:
-		if (!bytes)
-			bytes = insn->opnd_bytes;
-
+	case MMIO_WRITE_IMM:
 		memcpy(ghcb->shared_buffer, insn->immediate1.bytes, bytes);
-
 		ret = vc_do_mmio(ghcb, ctxt, bytes, false);
 		break;
-
-		/* MMIO Read */
-	case 0x8a:
-		bytes = 1;
-		fallthrough;
-	case 0x8b:
-		if (!bytes)
-			bytes = insn->opnd_bytes;
-
+	case MMIO_READ:
 		ret = vc_do_mmio(ghcb, ctxt, bytes, true);
 		if (ret)
 			break;
 
-		reg_data = vc_insn_get_reg(ctxt);
-		if (!reg_data)
-			return ES_DECODE_FAILED;
-
 		/* Zero-extend for 32-bit operation */
 		if (bytes == 4)
 			*reg_data = 0;
 
 		memcpy(reg_data, ghcb->shared_buffer, bytes);
 		break;
+	case MMIO_READ_ZERO_EXTEND:
+		ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+		if (ret)
+			break;
+
+		memset(reg_data, 0, insn->opnd_bytes);
+		memcpy(reg_data, ghcb->shared_buffer, bytes);
+		break;
+	case MMIO_READ_SIGN_EXTEND:
+		ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+		if (ret)
+			break;
 
-		/* MOVS instruction */
-	case 0xa4:
-		bytes = 1;
-		fallthrough;
-	case 0xa5:
-		if (!bytes)
-			bytes = insn->opnd_bytes;
+		if (bytes == 1) {
+			u8 *val = (u8 *)ghcb->shared_buffer;
 
-		ret = vc_handle_mmio_movs(ctxt, bytes);
+			sign_byte = (*val & 0x80) ? 0xff : 0x00;
+		} else {
+			u16 *val = (u16 *)ghcb->shared_buffer;
+
+			sign_byte = (*val & 0x8000) ? 0xff : 0x00;
+		}
+
+		/* Sign extend based on operand size */
+		memset(reg_data, sign_byte, insn->opnd_bytes);
+		memcpy(reg_data, ghcb->shared_buffer, bytes);
 		break;
-		/* Two-Byte Opcodes */
-	case 0x0f:
-		ret = vc_handle_mmio_twobyte_ops(ghcb, ctxt);
+	case MMIO_MOVS:
+		ret = vc_handle_mmio_movs(ctxt, bytes);
 		break;
 	default:
 		ret = ES_UNSUPPORTED;
+		break;
 	}
 
 	return ret;
-- 
2.32.0