[PATCH V5 0/3] Vm_event memory introspection helpers

[PATCH V5 0/3] Vm_event memory introspection helpers

[Razvan Cojocaru]

This series addresses V4 reviews (patch 1/3), and has been
rebased to compile on top of Xen staging (which now includes
a few patches from Tamas that prevent V4 from compiling).

I've also moved x86 logic in patch 3/3 to x86 source files, this
seems to have gone unnoticed but would likely have not compiled
on ARM.

I've removed George's ack from patch 1/3, but it most likely
only pertained to the mm bits which have remained unchanged.
George, if you stand by your ack with V5 then please re-add it,
thanks!

Patches 2/3 and 3/3 do have acks.

[PATCH V5 1/3] xen/mem_access: Support for memory-content hiding
[PATCH V5 2/3] xen/vm_event: Support for guest-requested events
[PATCH V5 3/3] xen/vm_event: Deny register writes if refused by
vm_event reply


Thanks in advance for your reviews,
Razvan

[PATCH V5 1/3] xen/mem_access: Support for memory-content hiding

[Razvan Cojocaru]

This patch adds support for memory-content hiding, by modifying the
value returned by emulated instructions that read certain memory
addresses that contain sensitive data. The patch only applies to
cases where VM_FLAG_ACCESS_EMULATE has been set to a vm_event
response.

Signed-off-by: Razvan Cojocaru <rcojocaru@bitdefender.com>

---
Changes since V4:
 - Rebased the patch to take into account Tamas' "x86/vm_event:
   toggle singlestep from vm_event response".
 - Renamed MEM_ACCESS_EMULATE to VM_EVENT_FLAG_EMULATE in stale
   comment in vm_event.h.
 - Factored out common code setting the read data in emulate.c.
 - Made sure the x86-specific code doesn't leak in the common
   code (added two new functions to the x86 and ARM vm_event.c
   files).
---
 tools/tests/xen-access/xen-access.c |    2 +-
 xen/arch/x86/domain.c               |    2 +
 xen/arch/x86/hvm/emulate.c          |  106 ++++++++++++++++++++++++++++++++++-
 xen/arch/x86/hvm/event.c            |   50 ++++++++---------
 xen/arch/x86/mm/p2m.c               |   92 +++++++++++++++++-------------
 xen/arch/x86/vm_event.c             |   30 ++++++++++
 xen/common/vm_event.c               |    8 +++
 xen/include/asm-arm/vm_event.h      |   13 +++++
 xen/include/asm-x86/domain.h        |    1 +
 xen/include/asm-x86/hvm/emulate.h   |   10 +++-
 xen/include/asm-x86/vm_event.h      |    4 ++
 xen/include/public/vm_event.h       |   35 +++++++++---
 12 files changed, 275 insertions(+), 78 deletions(-)

diff --git a/tools/tests/xen-access/xen-access.c b/tools/tests/xen-access/xen-access.c
index 12ab921..e6ca9ba 100644
--- a/tools/tests/xen-access/xen-access.c
+++ b/tools/tests/xen-access/xen-access.c
@@ -530,7 +530,7 @@ int main(int argc, char *argv[])
                 break;
             case VM_EVENT_REASON_SOFTWARE_BREAKPOINT:
                 printf("Breakpoint: rip=%016"PRIx64", gfn=%"PRIx64" (vcpu %d)\n",
-                       req.regs.x86.rip,
+                       req.data.regs.x86.rip,
                        req.u.software_breakpoint.gfn,
                        req.vcpu_id);
 
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 34ecd7c..6596408 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -511,6 +511,8 @@ int vcpu_initialise(struct vcpu *v)
 
 void vcpu_destroy(struct vcpu *v)
 {
+    xfree(v->arch.vm_event.emul_read_data);
+
     if ( is_pv_32bit_vcpu(v) )
     {
         free_compat_arg_xlat(v);
diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
index 795321c..30def25 100644
--- a/xen/arch/x86/hvm/emulate.c
+++ b/xen/arch/x86/hvm/emulate.c
@@ -67,6 +67,27 @@ static int null_write(const struct hvm_io_handler *handler,
     return X86EMUL_OKAY;
 }
 
+static int set_context_data(void *buffer, unsigned int bytes)
+{
+    struct vcpu *curr = current;
+
+    if ( !curr->arch.vm_event.emul_read_data )
+        return X86EMUL_UNHANDLEABLE;
+    else
+    {
+        unsigned int safe_bytes =
+            min(bytes, curr->arch.vm_event.emul_read_data->size);
+
+        if ( safe_bytes )
+            memcpy(buffer, curr->arch.vm_event.emul_read_data->data,
+                   safe_bytes);
+
+        memset(buffer + safe_bytes, 0, bytes - safe_bytes);
+    }
+
+    return X86EMUL_OKAY;
+}
+
 static const struct hvm_io_ops null_ops = {
     .read = null_read,
     .write = null_write
@@ -771,6 +792,12 @@ static int hvmemul_read(
     unsigned int bytes,
     struct x86_emulate_ctxt *ctxt)
 {
+    struct hvm_emulate_ctxt *hvmemul_ctxt =
+        container_of(ctxt, struct hvm_emulate_ctxt, ctxt);
+
+    if ( unlikely(hvmemul_ctxt->set_context) )
+        return set_context_data(p_data, bytes);
+
     return __hvmemul_read(
         seg, offset, p_data, bytes, hvm_access_read,
         container_of(ctxt, struct hvm_emulate_ctxt, ctxt));
@@ -963,6 +990,17 @@ static int hvmemul_cmpxchg(
     unsigned int bytes,
     struct x86_emulate_ctxt *ctxt)
 {
+    struct hvm_emulate_ctxt *hvmemul_ctxt =
+        container_of(ctxt, struct hvm_emulate_ctxt, ctxt);
+
+    if ( unlikely(hvmemul_ctxt->set_context) )
+    {
+        int rc = set_context_data(p_new, bytes);
+
+        if ( rc != X86EMUL_OKAY )
+            return rc;
+    }
+
     /* Fix this in case the guest is really relying on r-m-w atomicity. */
     return hvmemul_write(seg, offset, p_new, bytes, ctxt);
 }
@@ -1005,6 +1043,36 @@ static int hvmemul_rep_ins(
                                !!(ctxt->regs->eflags & X86_EFLAGS_DF), gpa);
 }
 
+static int hvmemul_rep_outs_set_context(
+    enum x86_segment src_seg,
+    unsigned long src_offset,
+    uint16_t dst_port,
+    unsigned int bytes_per_rep,
+    unsigned long *reps,
+    struct x86_emulate_ctxt *ctxt)
+{
+    unsigned int bytes = *reps * bytes_per_rep;
+    char *buf;
+    int rc;
+
+    buf = xmalloc_array(char, bytes);
+
+    if ( buf == NULL )
+        return X86EMUL_UNHANDLEABLE;
+
+    rc = set_context_data(buf, bytes);
+
+    if ( rc != X86EMUL_OKAY )
+        goto out;
+
+    rc = hvmemul_do_pio_buffer(dst_port, bytes, IOREQ_WRITE, buf);
+
+out:
+    xfree(buf);
+
+    return rc;
+}
+
 static int hvmemul_rep_outs(
     enum x86_segment src_seg,
     unsigned long src_offset,
@@ -1021,6 +1089,10 @@ static int hvmemul_rep_outs(
     p2m_type_t p2mt;
     int rc;
 
+    if ( unlikely(hvmemul_ctxt->set_context) )
+        return hvmemul_rep_outs_set_context(src_seg, src_offset, dst_port,
+                                            bytes_per_rep, reps, ctxt);
+
     rc = hvmemul_virtual_to_linear(
         src_seg, src_offset, bytes_per_rep, reps, hvm_access_read,
         hvmemul_ctxt, &addr);
@@ -1133,7 +1205,20 @@ static int hvmemul_rep_movs(
      */
     rc = hvm_copy_from_guest_phys(buf, sgpa, bytes);
     if ( rc == HVMCOPY_okay )
+    {
+        if ( unlikely(hvmemul_ctxt->set_context) )
+        {
+            rc = set_context_data(buf, bytes);
+
+            if ( rc != X86EMUL_OKAY)
+            {
+                xfree(buf);
+                return rc;
+            }
+        }
+
         rc = hvm_copy_to_guest_phys(dgpa, buf, bytes);
+    }
 
     xfree(buf);
 
@@ -1292,7 +1377,14 @@ static int hvmemul_read_io(
     unsigned long *val,
     struct x86_emulate_ctxt *ctxt)
 {
+    struct hvm_emulate_ctxt *hvmemul_ctxt =
+        container_of(ctxt, struct hvm_emulate_ctxt, ctxt);
+
     *val = 0;
+
+    if ( unlikely(hvmemul_ctxt->set_context) )
+        return set_context_data(val, bytes);
+
     return hvmemul_do_pio_buffer(port, bytes, IOREQ_READ, val);
 }
 
@@ -1677,7 +1769,7 @@ int hvm_emulate_one_no_write(
     return _hvm_emulate_one(hvmemul_ctxt, &hvm_emulate_ops_no_write);
 }
 
-void hvm_mem_access_emulate_one(bool_t nowrite, unsigned int trapnr,
+void hvm_mem_access_emulate_one(enum emul_kind kind, unsigned int trapnr,
     unsigned int errcode)
 {
     struct hvm_emulate_ctxt ctx = {{ 0 }};
@@ -1685,10 +1777,17 @@ void hvm_mem_access_emulate_one(bool_t nowrite, unsigned int trapnr,
 
     hvm_emulate_prepare(&ctx, guest_cpu_user_regs());
 
-    if ( nowrite )
+    switch ( kind )
+    {
+    case EMUL_KIND_NOWRITE:
         rc = hvm_emulate_one_no_write(&ctx);
-    else
+        break;
+    case EMUL_KIND_SET_CONTEXT:
+        ctx.set_context = 1;
+        /* Intentional fall-through. */
+    default:
         rc = hvm_emulate_one(&ctx);
+    }
 
     switch ( rc )
     {
@@ -1722,6 +1821,7 @@ void hvm_emulate_prepare(
     hvmemul_ctxt->ctxt.force_writeback = 1;
     hvmemul_ctxt->seg_reg_accessed = 0;
     hvmemul_ctxt->seg_reg_dirty = 0;
+    hvmemul_ctxt->set_context = 0;
     hvmemul_get_seg_reg(x86_seg_cs, hvmemul_ctxt);
     hvmemul_get_seg_reg(x86_seg_ss, hvmemul_ctxt);
 }
diff --git a/xen/arch/x86/hvm/event.c b/xen/arch/x86/hvm/event.c
index 53b9ca4..5341937 100644
--- a/xen/arch/x86/hvm/event.c
+++ b/xen/arch/x86/hvm/event.c
@@ -30,31 +30,31 @@ static void hvm_event_fill_regs(vm_event_request_t *req)
     const struct cpu_user_regs *regs = guest_cpu_user_regs();
     const struct vcpu *curr = current;
 
-    req->regs.x86.rax = regs->eax;
-    req->regs.x86.rcx = regs->ecx;
-    req->regs.x86.rdx = regs->edx;
-    req->regs.x86.rbx = regs->ebx;
-    req->regs.x86.rsp = regs->esp;
-    req->regs.x86.rbp = regs->ebp;
-    req->regs.x86.rsi = regs->esi;
-    req->regs.x86.rdi = regs->edi;
-
-    req->regs.x86.r8  = regs->r8;
-    req->regs.x86.r9  = regs->r9;
-    req->regs.x86.r10 = regs->r10;
-    req->regs.x86.r11 = regs->r11;
-    req->regs.x86.r12 = regs->r12;
-    req->regs.x86.r13 = regs->r13;
-    req->regs.x86.r14 = regs->r14;
-    req->regs.x86.r15 = regs->r15;
-
-    req->regs.x86.rflags = regs->eflags;
-    req->regs.x86.rip    = regs->eip;
-
-    req->regs.x86.msr_efer = curr->arch.hvm_vcpu.guest_efer;
-    req->regs.x86.cr0 = curr->arch.hvm_vcpu.guest_cr[0];
-    req->regs.x86.cr3 = curr->arch.hvm_vcpu.guest_cr[3];
-    req->regs.x86.cr4 = curr->arch.hvm_vcpu.guest_cr[4];
+    req->data.regs.x86.rax = regs->eax;
+    req->data.regs.x86.rcx = regs->ecx;
+    req->data.regs.x86.rdx = regs->edx;
+    req->data.regs.x86.rbx = regs->ebx;
+    req->data.regs.x86.rsp = regs->esp;
+    req->data.regs.x86.rbp = regs->ebp;
+    req->data.regs.x86.rsi = regs->esi;
+    req->data.regs.x86.rdi = regs->edi;
+
+    req->data.regs.x86.r8  = regs->r8;
+    req->data.regs.x86.r9  = regs->r9;
+    req->data.regs.x86.r10 = regs->r10;
+    req->data.regs.x86.r11 = regs->r11;
+    req->data.regs.x86.r12 = regs->r12;
+    req->data.regs.x86.r13 = regs->r13;
+    req->data.regs.x86.r14 = regs->r14;
+    req->data.regs.x86.r15 = regs->r15;
+
+    req->data.regs.x86.rflags = regs->eflags;
+    req->data.regs.x86.rip    = regs->eip;
+
+    req->data.regs.x86.msr_efer = curr->arch.hvm_vcpu.guest_efer;
+    req->data.regs.x86.cr0 = curr->arch.hvm_vcpu.guest_cr[0];
+    req->data.regs.x86.cr3 = curr->arch.hvm_vcpu.guest_cr[3];
+    req->data.regs.x86.cr4 = curr->arch.hvm_vcpu.guest_cr[4];
 }
 
 static int hvm_event_traps(uint8_t sync, vm_event_request_t *req)
diff --git a/xen/arch/x86/mm/p2m.c b/xen/arch/x86/mm/p2m.c
index f180cee..6fe6387 100644
--- a/xen/arch/x86/mm/p2m.c
+++ b/xen/arch/x86/mm/p2m.c
@@ -1369,49 +1369,49 @@ static void p2m_vm_event_fill_regs(vm_event_request_t *req)
     /* Architecture-specific vmcs/vmcb bits */
     hvm_funcs.save_cpu_ctxt(curr, &ctxt);
 
-    req->regs.x86.rax = regs->eax;
-    req->regs.x86.rcx = regs->ecx;
-    req->regs.x86.rdx = regs->edx;
-    req->regs.x86.rbx = regs->ebx;
-    req->regs.x86.rsp = regs->esp;
-    req->regs.x86.rbp = regs->ebp;
-    req->regs.x86.rsi = regs->esi;
-    req->regs.x86.rdi = regs->edi;
-
-    req->regs.x86.r8  = regs->r8;
-    req->regs.x86.r9  = regs->r9;
-    req->regs.x86.r10 = regs->r10;
-    req->regs.x86.r11 = regs->r11;
-    req->regs.x86.r12 = regs->r12;
-    req->regs.x86.r13 = regs->r13;
-    req->regs.x86.r14 = regs->r14;
-    req->regs.x86.r15 = regs->r15;
-
-    req->regs.x86.rflags = regs->eflags;
-    req->regs.x86.rip    = regs->eip;
-
-    req->regs.x86.dr7 = curr->arch.debugreg[7];
-    req->regs.x86.cr0 = ctxt.cr0;
-    req->regs.x86.cr2 = ctxt.cr2;
-    req->regs.x86.cr3 = ctxt.cr3;
-    req->regs.x86.cr4 = ctxt.cr4;
-
-    req->regs.x86.sysenter_cs = ctxt.sysenter_cs;
-    req->regs.x86.sysenter_esp = ctxt.sysenter_esp;
-    req->regs.x86.sysenter_eip = ctxt.sysenter_eip;
-
-    req->regs.x86.msr_efer = ctxt.msr_efer;
-    req->regs.x86.msr_star = ctxt.msr_star;
-    req->regs.x86.msr_lstar = ctxt.msr_lstar;
+    req->data.regs.x86.rax = regs->eax;
+    req->data.regs.x86.rcx = regs->ecx;
+    req->data.regs.x86.rdx = regs->edx;
+    req->data.regs.x86.rbx = regs->ebx;
+    req->data.regs.x86.rsp = regs->esp;
+    req->data.regs.x86.rbp = regs->ebp;
+    req->data.regs.x86.rsi = regs->esi;
+    req->data.regs.x86.rdi = regs->edi;
+
+    req->data.regs.x86.r8  = regs->r8;
+    req->data.regs.x86.r9  = regs->r9;
+    req->data.regs.x86.r10 = regs->r10;
+    req->data.regs.x86.r11 = regs->r11;
+    req->data.regs.x86.r12 = regs->r12;
+    req->data.regs.x86.r13 = regs->r13;
+    req->data.regs.x86.r14 = regs->r14;
+    req->data.regs.x86.r15 = regs->r15;
+
+    req->data.regs.x86.rflags = regs->eflags;
+    req->data.regs.x86.rip    = regs->eip;
+
+    req->data.regs.x86.dr7 = curr->arch.debugreg[7];
+    req->data.regs.x86.cr0 = ctxt.cr0;
+    req->data.regs.x86.cr2 = ctxt.cr2;
+    req->data.regs.x86.cr3 = ctxt.cr3;
+    req->data.regs.x86.cr4 = ctxt.cr4;
+
+    req->data.regs.x86.sysenter_cs = ctxt.sysenter_cs;
+    req->data.regs.x86.sysenter_esp = ctxt.sysenter_esp;
+    req->data.regs.x86.sysenter_eip = ctxt.sysenter_eip;
+
+    req->data.regs.x86.msr_efer = ctxt.msr_efer;
+    req->data.regs.x86.msr_star = ctxt.msr_star;
+    req->data.regs.x86.msr_lstar = ctxt.msr_lstar;
 
     hvm_get_segment_register(curr, x86_seg_fs, &seg);
-    req->regs.x86.fs_base = seg.base;
+    req->data.regs.x86.fs_base = seg.base;
 
     hvm_get_segment_register(curr, x86_seg_gs, &seg);
-    req->regs.x86.gs_base = seg.base;
+    req->data.regs.x86.gs_base = seg.base;
 
     hvm_get_segment_register(curr, x86_seg_cs, &seg);
-    req->regs.x86.cs_arbytes = seg.attr.bytes;
+    req->data.regs.x86.cs_arbytes = seg.attr.bytes;
 }
 
 void p2m_mem_access_emulate_check(struct vcpu *v,
@@ -1466,6 +1466,10 @@ void p2m_mem_access_emulate_check(struct vcpu *v,
         }
 
         v->arch.vm_event.emulate_flags = violation ? rsp->flags : 0;
+
+        if ( (rsp->flags & VM_EVENT_FLAG_SET_EMUL_READ_DATA) &&
+             v->arch.vm_event.emul_read_data )
+            *v->arch.vm_event.emul_read_data = rsp->data.emul_read_data;
     }
 }
 
@@ -1552,9 +1556,17 @@ bool_t p2m_mem_access_check(paddr_t gpa, unsigned long gla,
 
     if ( v->arch.vm_event.emulate_flags )
     {
-        hvm_mem_access_emulate_one((v->arch.vm_event.emulate_flags &
-                                    VM_EVENT_FLAG_EMULATE_NOWRITE) != 0,
-                                   TRAP_invalid_op, HVM_DELIVER_NO_ERROR_CODE);
+        enum emul_kind kind = EMUL_KIND_NORMAL;
+
+        if ( v->arch.vm_event.emulate_flags &
+             VM_EVENT_FLAG_SET_EMUL_READ_DATA )
+            kind = EMUL_KIND_SET_CONTEXT;
+        else if ( v->arch.vm_event.emulate_flags &
+                  VM_EVENT_FLAG_EMULATE_NOWRITE )
+            kind = EMUL_KIND_NOWRITE;
+
+        hvm_mem_access_emulate_one(kind, TRAP_invalid_op,
+                                   HVM_DELIVER_NO_ERROR_CODE);
 
         v->arch.vm_event.emulate_flags = 0;
         return 1;
diff --git a/xen/arch/x86/vm_event.c b/xen/arch/x86/vm_event.c
index c390225..0515161 100644
--- a/xen/arch/x86/vm_event.c
+++ b/xen/arch/x86/vm_event.c
@@ -23,6 +23,36 @@
 #include <xen/sched.h>
 #include <asm/hvm/hvm.h>
 
+int vm_event_init_domain(struct domain *d)
+{
+    struct vcpu *v;
+
+    for_each_vcpu( d, v )
+    {
+        if ( v->arch.vm_event.emul_read_data )
+            continue;
+
+        v->arch.vm_event.emul_read_data =
+            xzalloc(struct vm_event_emul_read_data);
+
+        if ( !v->arch.vm_event.emul_read_data )
+            return -ENOMEM;
+    }
+
+    return 0;
+}
+
+void vm_event_cleanup_domain(struct domain *d)
+{
+    struct vcpu *v;
+
+    for_each_vcpu( d, v )
+    {
+        xfree(v->arch.vm_event.emul_read_data);
+        v->arch.vm_event.emul_read_data = NULL;
+    }
+}
+
 void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v)
 {
     if ( !is_hvm_domain(d) || !atomic_read(&v->vm_event_pause_count) )
diff --git a/xen/common/vm_event.c b/xen/common/vm_event.c
index a4b9c36..0007d70 100644
--- a/xen/common/vm_event.c
+++ b/xen/common/vm_event.c
@@ -64,6 +64,11 @@ static int vm_event_enable(
     vm_event_ring_lock_init(ved);
     vm_event_ring_lock(ved);
 
+    rc = vm_event_init_domain(d);
+
+    if ( rc < 0 )
+        goto err;
+
     rc = prepare_ring_for_helper(d, ring_gfn, &ved->ring_pg_struct,
                                     &ved->ring_page);
     if ( rc < 0 )
@@ -226,6 +231,9 @@ static int vm_event_disable(struct domain *d, struct vm_event_domain *ved)
 
         destroy_ring_for_helper(&ved->ring_page,
                                 ved->ring_pg_struct);
+
+        vm_event_cleanup_domain(d);
+
         vm_event_ring_unlock(ved);
     }
 
diff --git a/xen/include/asm-arm/vm_event.h b/xen/include/asm-arm/vm_event.h
index a517495..20469a8 100644
--- a/xen/include/asm-arm/vm_event.h
+++ b/xen/include/asm-arm/vm_event.h
@@ -23,6 +23,19 @@
 #include <xen/sched.h>
 
 static inline
+int vm_event_init_domain(struct domain *d)
+{
+    /* Not supported on ARM. */
+    return 0;
+}
+
+static inline
+void vm_event_cleanup_domain(struct domain *d)
+{
+    /* Not supported on ARM. */
+}
+
+static inline
 void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v)
 {
     /* Not supported on ARM. */
diff --git a/xen/include/asm-x86/domain.h b/xen/include/asm-x86/domain.h
index 201436d..c5ad1cb 100644
--- a/xen/include/asm-x86/domain.h
+++ b/xen/include/asm-x86/domain.h
@@ -512,6 +512,7 @@ struct arch_vcpu
         uint32_t emulate_flags;
         unsigned long gpa;
         unsigned long eip;
+        struct vm_event_emul_read_data *emul_read_data;
     } vm_event;
 
 };
diff --git a/xen/include/asm-x86/hvm/emulate.h b/xen/include/asm-x86/hvm/emulate.h
index 594be38..49134b5 100644
--- a/xen/include/asm-x86/hvm/emulate.h
+++ b/xen/include/asm-x86/hvm/emulate.h
@@ -32,13 +32,21 @@ struct hvm_emulate_ctxt {
     struct hvm_trap trap;
 
     uint32_t intr_shadow;
+
+    bool_t set_context;
+};
+
+enum emul_kind {
+    EMUL_KIND_NORMAL,
+    EMUL_KIND_NOWRITE,
+    EMUL_KIND_SET_CONTEXT
 };
 
 int hvm_emulate_one(
     struct hvm_emulate_ctxt *hvmemul_ctxt);
 int hvm_emulate_one_no_write(
     struct hvm_emulate_ctxt *hvmemul_ctxt);
-void hvm_mem_access_emulate_one(bool_t nowrite,
+void hvm_mem_access_emulate_one(enum emul_kind kind,
     unsigned int trapnr,
     unsigned int errcode);
 void hvm_emulate_prepare(
diff --git a/xen/include/asm-x86/vm_event.h b/xen/include/asm-x86/vm_event.h
index 7cc3a3d..3881783 100644
--- a/xen/include/asm-x86/vm_event.h
+++ b/xen/include/asm-x86/vm_event.h
@@ -22,6 +22,10 @@
 
 #include <xen/sched.h>
 
+int vm_event_init_domain(struct domain *d);
+
+void vm_event_cleanup_domain(struct domain *d);
+
 void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v);
 
 #endif /* __ASM_X86_VM_EVENT_H__ */
diff --git a/xen/include/public/vm_event.h b/xen/include/public/vm_event.h
index c756c7c..4d89c38 100644
--- a/xen/include/public/vm_event.h
+++ b/xen/include/public/vm_event.h
@@ -44,9 +44,9 @@
  *  paused
  * VCPU_PAUSED in a response signals to unpause the vCPU
  */
-#define VM_EVENT_FLAG_VCPU_PAUSED       (1 << 0)
+#define VM_EVENT_FLAG_VCPU_PAUSED        (1 << 0)
 /* Flags to aid debugging vm_event */
-#define VM_EVENT_FLAG_FOREIGN           (1 << 1)
+#define VM_EVENT_FLAG_FOREIGN            (1 << 1)
 /*
  * The following flags can be set in response to a mem_access event.
  *
@@ -54,17 +54,26 @@
  * This will allow the guest to continue execution without lifting the page
  * access restrictions.
  */
-#define VM_EVENT_FLAG_EMULATE           (1 << 2)
+#define VM_EVENT_FLAG_EMULATE            (1 << 2)
 /*
- * Same as MEM_ACCESS_EMULATE, but with write operations or operations
+ * Same as VM_EVENT_FLAG_EMULATE, but with write operations or operations
  * potentially having side effects (like memory mapped or port I/O) disabled.
  */
-#define VM_EVENT_FLAG_EMULATE_NOWRITE   (1 << 3)
+#define VM_EVENT_FLAG_EMULATE_NOWRITE    (1 << 3)
 /*
  * Toggle singlestepping on vm_event response.
  * Requires the vCPU to be paused already (synchronous events only).
  */
-#define VM_EVENT_FLAG_TOGGLE_SINGLESTEP (1 << 4)
+#define VM_EVENT_FLAG_TOGGLE_SINGLESTEP  (1 << 4)
+/*
+ * Data is being sent back to the hypervisor in the event response, to be
+ * returned by the read function when emulating an instruction.
+ * This flag is only useful when combined with VM_EVENT_FLAG_EMULATE
+ * and takes precedence if combined with VM_EVENT_FLAG_EMULATE_NOWRITE
+ * (i.e. if both VM_EVENT_FLAG_EMULATE_NOWRITE and
+ * VM_EVENT_FLAG_SET_EMUL_READ_DATA are set, only the latter will be honored).
+ */
+#define VM_EVENT_FLAG_SET_EMUL_READ_DATA (1 << 5)
 
 /*
  * Reasons for the vm event request
@@ -194,6 +203,12 @@ struct vm_event_sharing {
     uint32_t _pad;
 };
 
+struct vm_event_emul_read_data {
+    uint32_t size;
+    /* The struct is used in a union with vm_event_regs_x86. */
+    uint8_t  data[sizeof(struct vm_event_regs_x86) - sizeof(uint32_t)];
+};
+
 typedef struct vm_event_st {
     uint32_t version;   /* VM_EVENT_INTERFACE_VERSION */
     uint32_t flags;     /* VM_EVENT_FLAG_* */
@@ -211,8 +226,12 @@ typedef struct vm_event_st {
     } u;
 
     union {
-        struct vm_event_regs_x86 x86;
-    } regs;
+        union {
+            struct vm_event_regs_x86 x86;
+        } regs;
+
+        struct vm_event_emul_read_data emul_read_data;
+    } data;
 } vm_event_request_t, vm_event_response_t;
 
 DEFINE_RING_TYPES(vm_event, vm_event_request_t, vm_event_response_t);
-- 
1.7.9.5

Re: [PATCH V5 1/3] xen/mem_access: Support for memory-content hiding

[Lengyel, Tamas]

[-- Attachment #1.1: Type: text/plain, Size: 510 bytes --]

On Mon, Jul 13, 2015 at 1:14 PM, Razvan Cojocaru <rcojocaru@bitdefender.com>
wrote:

> This patch adds support for memory-content hiding, by modifying the
> value returned by emulated instructions that read certain memory
> addresses that contain sensitive data. The patch only applies to
> cases where VM_FLAG_ACCESS_EMULATE has been set to a vm_event
> response.
>
> Signed-off-by: Razvan Cojocaru <rcojocaru@bitdefender.com>
>

You seem to have missed my Acked-by I gave on v4 of this patch.

Cheers,
Tamas

[-- Attachment #1.2: Type: text/html, Size: 991 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH V5 1/3] xen/mem_access: Support for memory-content hiding

[Razvan Cojocaru]

On 07/13/2015 08:32 PM, Lengyel, Tamas wrote:
> 
> 
> On Mon, Jul 13, 2015 at 1:14 PM, Razvan Cojocaru
> <rcojocaru@bitdefender.com <mailto:rcojocaru@bitdefender.com>> wrote:
> 
>     This patch adds support for memory-content hiding, by modifying the
>     value returned by emulated instructions that read certain memory
>     addresses that contain sensitive data. The patch only applies to
>     cases where VM_FLAG_ACCESS_EMULATE has been set to a vm_event
>     response.
> 
>     Signed-off-by: Razvan Cojocaru <rcojocaru@bitdefender.com
>     <mailto:rcojocaru@bitdefender.com>>
> 
> 
> You seem to have missed my Acked-by I gave on v4 of this patch.

Right, sorry :)

In this case there's your ack and possibly George's for 1/3, so all the
patches in the series have at least one ack.


Thanks,
Razvan

Re: [PATCH V5 1/3] xen/mem_access: Support for memory-content hiding

[Jan Beulich]

>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
> --- a/xen/arch/x86/domain.c
> +++ b/xen/arch/x86/domain.c
> @@ -511,6 +511,8 @@ int vcpu_initialise(struct vcpu *v)
>  
>  void vcpu_destroy(struct vcpu *v)
>  {
> +    xfree(v->arch.vm_event.emul_read_data);

Is this still needed now that you have
vm_event_cleanup_domain()?

> --- a/xen/arch/x86/hvm/emulate.c
> +++ b/xen/arch/x86/hvm/emulate.c
> @@ -67,6 +67,27 @@ static int null_write(const struct hvm_io_handler *handler,
>      return X86EMUL_OKAY;
>  }
>  
> +static int set_context_data(void *buffer, unsigned int bytes)

I think in the context of this function alone, "size" would be better
than "bytes".

> +{
> +    struct vcpu *curr = current;
> +
> +    if ( !curr->arch.vm_event.emul_read_data )
> +        return X86EMUL_UNHANDLEABLE;
> +    else

Else after the respective if() unconditionally returning is odd.
Perhaps better to invert the if() condition...

> +    {
> +        unsigned int safe_bytes =
> +            min(bytes, curr->arch.vm_event.emul_read_data->size);
> +
> +        if ( safe_bytes )
> +            memcpy(buffer, curr->arch.vm_event.emul_read_data->data,
> +                   safe_bytes);

So why did you still keep this conditional?

> @@ -1005,6 +1043,36 @@ static int hvmemul_rep_ins(
>                                 !!(ctxt->regs->eflags & X86_EFLAGS_DF), gpa);
>  }
>  
> +static int hvmemul_rep_outs_set_context(
> +    enum x86_segment src_seg,
> +    unsigned long src_offset,
> +    uint16_t dst_port,
> +    unsigned int bytes_per_rep,
> +    unsigned long *reps,
> +    struct x86_emulate_ctxt *ctxt)
> +{
> +    unsigned int bytes = *reps * bytes_per_rep;
> +    char *buf;
> +    int rc;
> +
> +    buf = xmalloc_array(char, bytes);
> +
> +    if ( buf == NULL )
> +        return X86EMUL_UNHANDLEABLE;
> +
> +    rc = set_context_data(buf, bytes);
> +
> +    if ( rc != X86EMUL_OKAY )
> +        goto out;
> +
> +    rc = hvmemul_do_pio_buffer(dst_port, bytes, IOREQ_WRITE, buf);
> +
> +out:

Labels should be indented by at least one space. But realistically
the code wouldn't look worse without any goto...

> @@ -1133,7 +1205,20 @@ static int hvmemul_rep_movs(
>       */
>      rc = hvm_copy_from_guest_phys(buf, sgpa, bytes);
>      if ( rc == HVMCOPY_okay )
> +    {
> +        if ( unlikely(hvmemul_ctxt->set_context) )
> +        {
> +            rc = set_context_data(buf, bytes);
> +
> +            if ( rc != X86EMUL_OKAY)
> +            {
> +                xfree(buf);
> +                return rc;
> +            }
> +        }
> +
>          rc = hvm_copy_to_guest_phys(dgpa, buf, bytes);
> +    }

Why do you not bypass hvm_copy_from_guest_phys() here? This
way it would - afaict - become consistent with the other cases.

> --- a/xen/arch/x86/vm_event.c
> +++ b/xen/arch/x86/vm_event.c
> @@ -23,6 +23,36 @@
>  #include <xen/sched.h>
>  #include <asm/hvm/hvm.h>
>  
> +int vm_event_init_domain(struct domain *d)
> +{
> +    struct vcpu *v;
> +
> +    for_each_vcpu( d, v )

Either you consider for_each_vcpu a keyword-like thing, then this
is missing a blank before the opening parenthesis, or you don't, in
which case the blanks immediately inside the parentheses should
be dropped.

> +    {
> +        if ( v->arch.vm_event.emul_read_data )
> +            continue;
> +
> +        v->arch.vm_event.emul_read_data =
> +            xzalloc(struct vm_event_emul_read_data);
> +
> +        if ( !v->arch.vm_event.emul_read_data )
> +            return -ENOMEM;
> +    }
> +
> +    return 0;
> +}
> +
> +void vm_event_cleanup_domain(struct domain *d)
> +{
> +    struct vcpu *v;
> +
> +    for_each_vcpu( d, v )
> +    {
> +        xfree(v->arch.vm_event.emul_read_data);
> +        v->arch.vm_event.emul_read_data = NULL;
> +    }
> +}

There not being a race here is attributed to vm_event_enable()
being implicitly serialized by the domctl lock, and
vm_event_disable(), beyond its domctl use, only being on domain
cleanup paths? Would be nice to have a comment to that effect.

Jan

Re: [PATCH V5 1/3] xen/mem_access: Support for memory-content hiding

[Razvan Cojocaru]

On 07/14/2015 03:22 PM, Jan Beulich wrote:
>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>> --- a/xen/arch/x86/domain.c
>> +++ b/xen/arch/x86/domain.c
>> @@ -511,6 +511,8 @@ int vcpu_initialise(struct vcpu *v)
>>  
>>  void vcpu_destroy(struct vcpu *v)
>>  {
>> +    xfree(v->arch.vm_event.emul_read_data);
> 
> Is this still needed now that you have
> vm_event_cleanup_domain()?

I had thought that there might be a code path where
vm_event_cleanup_domain() doesn't get called and yet the domain is being
destroyed, but I can't find anything obvious in the code except a
comment in arch/x86/mm/shadow/common.c - shadow_final_teardown() -
stating that "It is possible for a domain that never got domain_kill()ed
to get here with its shadow allocation intact.".

Since common/domain.c's domain_kill() seems to be the only caller of
vm_event_cleanup(), I took that comment to mean that it could be
possible to end up in vcpu_destroy() without vm_event_cleanup_domain()
having been called, so I took the better-safe-than-sorry approach.

That is also why I'm setting v->arch.vm_event.emul_read_data to NULL in
the vm_event domain cleanup function, so that this xfree() does not
double-free.

But this xfree() is probably not needed, so if confirmed I'll remove it.

>> --- a/xen/arch/x86/hvm/emulate.c
>> +++ b/xen/arch/x86/hvm/emulate.c
>> @@ -67,6 +67,27 @@ static int null_write(const struct hvm_io_handler *handler,
>>      return X86EMUL_OKAY;
>>  }
>>  
>> +static int set_context_data(void *buffer, unsigned int bytes)
> 
> I think in the context of this function alone, "size" would be better
> than "bytes".

Ack.

>> +{
>> +    struct vcpu *curr = current;
>> +
>> +    if ( !curr->arch.vm_event.emul_read_data )
>> +        return X86EMUL_UNHANDLEABLE;
>> +    else
> 
> Else after the respective if() unconditionally returning is odd.
> Perhaps better to invert the if() condition...

I agree, I only used the else so that I wouldn't need to put safe_bytes
on the stack if I didn't have to. I'll invert the condition.

>> +    {
>> +        unsigned int safe_bytes =
>> +            min(bytes, curr->arch.vm_event.emul_read_data->size);
>> +
>> +        if ( safe_bytes )
>> +            memcpy(buffer, curr->arch.vm_event.emul_read_data->data,
>> +                   safe_bytes);
> 
> So why did you still keep this conditional?

I thought a memcpy() call that ends up doing nothing (copying 0 bytes)
would be expensive and I've tried to optimize the code by not doing the
call if safe_bytes == 0.

Since nobody else seems to think it's worth it, I'll remove it.

>> @@ -1005,6 +1043,36 @@ static int hvmemul_rep_ins(
>>                                 !!(ctxt->regs->eflags & X86_EFLAGS_DF), gpa);
>>  }
>>  
>> +static int hvmemul_rep_outs_set_context(
>> +    enum x86_segment src_seg,
>> +    unsigned long src_offset,
>> +    uint16_t dst_port,
>> +    unsigned int bytes_per_rep,
>> +    unsigned long *reps,
>> +    struct x86_emulate_ctxt *ctxt)
>> +{
>> +    unsigned int bytes = *reps * bytes_per_rep;
>> +    char *buf;
>> +    int rc;
>> +
>> +    buf = xmalloc_array(char, bytes);
>> +
>> +    if ( buf == NULL )
>> +        return X86EMUL_UNHANDLEABLE;
>> +
>> +    rc = set_context_data(buf, bytes);
>> +
>> +    if ( rc != X86EMUL_OKAY )
>> +        goto out;
>> +
>> +    rc = hvmemul_do_pio_buffer(dst_port, bytes, IOREQ_WRITE, buf);
>> +
>> +out:
> 
> Labels should be indented by at least one space. But realistically
> the code wouldn't look worse without any goto...

Understood, will remove the label completely.

>> @@ -1133,7 +1205,20 @@ static int hvmemul_rep_movs(
>>       */
>>      rc = hvm_copy_from_guest_phys(buf, sgpa, bytes);
>>      if ( rc == HVMCOPY_okay )
>> +    {
>> +        if ( unlikely(hvmemul_ctxt->set_context) )
>> +        {
>> +            rc = set_context_data(buf, bytes);
>> +
>> +            if ( rc != X86EMUL_OKAY)
>> +            {
>> +                xfree(buf);
>> +                return rc;
>> +            }
>> +        }
>> +
>>          rc = hvm_copy_to_guest_phys(dgpa, buf, bytes);
>> +    }
> 
> Why do you not bypass hvm_copy_from_guest_phys() here? This
> way it would - afaict - become consistent with the other cases.

You're right, it's unnecessary. Will remove the hvm_copy_from_guest_phys().

>> --- a/xen/arch/x86/vm_event.c
>> +++ b/xen/arch/x86/vm_event.c
>> @@ -23,6 +23,36 @@
>>  #include <xen/sched.h>
>>  #include <asm/hvm/hvm.h>
>>  
>> +int vm_event_init_domain(struct domain *d)
>> +{
>> +    struct vcpu *v;
>> +
>> +    for_each_vcpu( d, v )
> 
> Either you consider for_each_vcpu a keyword-like thing, then this
> is missing a blank before the opening parenthesis, or you don't, in
> which case the blanks immediately inside the parentheses should
> be dropped.

Understood, will fix it.

>> +    {
>> +        if ( v->arch.vm_event.emul_read_data )
>> +            continue;
>> +
>> +        v->arch.vm_event.emul_read_data =
>> +            xzalloc(struct vm_event_emul_read_data);
>> +
>> +        if ( !v->arch.vm_event.emul_read_data )
>> +            return -ENOMEM;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>> +void vm_event_cleanup_domain(struct domain *d)
>> +{
>> +    struct vcpu *v;
>> +
>> +    for_each_vcpu( d, v )
>> +    {
>> +        xfree(v->arch.vm_event.emul_read_data);
>> +        v->arch.vm_event.emul_read_data = NULL;
>> +    }
>> +}
> 
> There not being a race here is attributed to vm_event_enable()
> being implicitly serialized by the domctl lock, and
> vm_event_disable(), beyond its domctl use, only being on domain
> cleanup paths? Would be nice to have a comment to that effect.

Indeed, that's how I understood it to happen. I'll add a comment.


Thanks,
Razvan

Re: [PATCH V5 1/3] xen/mem_access: Support for memory-content hiding

[Jan Beulich]

>>> On 14.07.15 at 15:26, <rcojocaru@bitdefender.com> wrote:
> On 07/14/2015 03:22 PM, Jan Beulich wrote:
>>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>>> --- a/xen/arch/x86/domain.c
>>> +++ b/xen/arch/x86/domain.c
>>> @@ -511,6 +511,8 @@ int vcpu_initialise(struct vcpu *v)
>>>  
>>>  void vcpu_destroy(struct vcpu *v)
>>>  {
>>> +    xfree(v->arch.vm_event.emul_read_data);
>> 
>> Is this still needed now that you have
>> vm_event_cleanup_domain()?
> 
> I had thought that there might be a code path where
> vm_event_cleanup_domain() doesn't get called and yet the domain is being
> destroyed, but I can't find anything obvious in the code except a
> comment in arch/x86/mm/shadow/common.c - shadow_final_teardown() -
> stating that "It is possible for a domain that never got domain_kill()ed
> to get here with its shadow allocation intact.".

Tim?

> Since common/domain.c's domain_kill() seems to be the only caller of
> vm_event_cleanup(), I took that comment to mean that it could be
> possible to end up in vcpu_destroy() without vm_event_cleanup_domain()
> having been called, so I took the better-safe-than-sorry approach.

Better-safe-than-sorry would imply you'd also have to clear the
pointer in vcpu_destroy(), covering the (hypothetical?) case of
vm_event_cleanup_domain() being called afterwards.

>>> +    {
>>> +        unsigned int safe_bytes =
>>> +            min(bytes, curr->arch.vm_event.emul_read_data->size);
>>> +
>>> +        if ( safe_bytes )
>>> +            memcpy(buffer, curr->arch.vm_event.emul_read_data->data,
>>> +                   safe_bytes);
>> 
>> So why did you still keep this conditional?
> 
> I thought a memcpy() call that ends up doing nothing (copying 0 bytes)
> would be expensive and I've tried to optimize the code by not doing the
> call if safe_bytes == 0.

That argumentation would then also apply to the subsequent
memset().

> Since nobody else seems to think it's worth it, I'll remove it.

Thanks.

>>> @@ -1133,7 +1205,20 @@ static int hvmemul_rep_movs(
>>>       */
>>>      rc = hvm_copy_from_guest_phys(buf, sgpa, bytes);
>>>      if ( rc == HVMCOPY_okay )
>>> +    {
>>> +        if ( unlikely(hvmemul_ctxt->set_context) )
>>> +        {
>>> +            rc = set_context_data(buf, bytes);
>>> +
>>> +            if ( rc != X86EMUL_OKAY)
>>> +            {
>>> +                xfree(buf);
>>> +                return rc;
>>> +            }
>>> +        }
>>> +
>>>          rc = hvm_copy_to_guest_phys(dgpa, buf, bytes);
>>> +    }
>> 
>> Why do you not bypass hvm_copy_from_guest_phys() here? This
>> way it would - afaict - become consistent with the other cases.
> 
> You're right, it's unnecessary. Will remove the hvm_copy_from_guest_phys().

s/remove/bypass/ hopefully.

Jan

Re: [PATCH V5 1/3] xen/mem_access: Support for memory-content hiding

[Razvan Cojocaru]

On 07/14/2015 04:37 PM, Jan Beulich wrote:
>>>> On 14.07.15 at 15:26, <rcojocaru@bitdefender.com> wrote:
>> On 07/14/2015 03:22 PM, Jan Beulich wrote:
>>>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>>>> --- a/xen/arch/x86/domain.c
>>>> +++ b/xen/arch/x86/domain.c
>>>> @@ -511,6 +511,8 @@ int vcpu_initialise(struct vcpu *v)
>>>>  
>>>>  void vcpu_destroy(struct vcpu *v)
>>>>  {
>>>> +    xfree(v->arch.vm_event.emul_read_data);
>>>
>>> Is this still needed now that you have
>>> vm_event_cleanup_domain()?
>>
>> I had thought that there might be a code path where
>> vm_event_cleanup_domain() doesn't get called and yet the domain is being
>> destroyed, but I can't find anything obvious in the code except a
>> comment in arch/x86/mm/shadow/common.c - shadow_final_teardown() -
>> stating that "It is possible for a domain that never got domain_kill()ed
>> to get here with its shadow allocation intact.".
> 
> Tim?
> 
>> Since common/domain.c's domain_kill() seems to be the only caller of
>> vm_event_cleanup(), I took that comment to mean that it could be
>> possible to end up in vcpu_destroy() without vm_event_cleanup_domain()
>> having been called, so I took the better-safe-than-sorry approach.
> 
> Better-safe-than-sorry would imply you'd also have to clear the
> pointer in vcpu_destroy(), covering the (hypothetical?) case of
> vm_event_cleanup_domain() being called afterwards.

True. If that can happen I'll clear the pointer there too.

>>>> +    {
>>>> +        unsigned int safe_bytes =
>>>> +            min(bytes, curr->arch.vm_event.emul_read_data->size);
>>>> +
>>>> +        if ( safe_bytes )
>>>> +            memcpy(buffer, curr->arch.vm_event.emul_read_data->data,
>>>> +                   safe_bytes);
>>>
>>> So why did you still keep this conditional?
>>
>> I thought a memcpy() call that ends up doing nothing (copying 0 bytes)
>> would be expensive and I've tried to optimize the code by not doing the
>> call if safe_bytes == 0.
> 
> That argumentation would then also apply to the subsequent
> memset().
> 
>> Since nobody else seems to think it's worth it, I'll remove it.
> 
> Thanks.
> 
>>>> @@ -1133,7 +1205,20 @@ static int hvmemul_rep_movs(
>>>>       */
>>>>      rc = hvm_copy_from_guest_phys(buf, sgpa, bytes);
>>>>      if ( rc == HVMCOPY_okay )
>>>> +    {
>>>> +        if ( unlikely(hvmemul_ctxt->set_context) )
>>>> +        {
>>>> +            rc = set_context_data(buf, bytes);
>>>> +
>>>> +            if ( rc != X86EMUL_OKAY)
>>>> +            {
>>>> +                xfree(buf);
>>>> +                return rc;
>>>> +            }
>>>> +        }
>>>> +
>>>>          rc = hvm_copy_to_guest_phys(dgpa, buf, bytes);
>>>> +    }
>>>
>>> Why do you not bypass hvm_copy_from_guest_phys() here? This
>>> way it would - afaict - become consistent with the other cases.
>>
>> You're right, it's unnecessary. Will remove the hvm_copy_from_guest_phys().
> 
> s/remove/bypass/ hopefully.

Yes, sorry for the typo.


Thanks,
Razvan

[PATCH V5 2/3] xen/vm_event: Support for guest-requested events

[Razvan Cojocaru]

Added support for a new class of vm_events: VM_EVENT_REASON_REQUEST,
sent via HVMOP_request_vm_event. The guest can request that a
generic vm_event (containing only the vm_event-filled guest registers
as information) be sent to userspace by setting up the correct
registers and doing a VMCALL. For example, for a 32-bit guest, this
means: EAX = 34 (hvmop), EBX = 24 (HVMOP_guest_request_vm_event),
ECX = 0 (NULL required for the hypercall parameter, reserved).

Signed-off-by: Razvan Cojocaru <rcojocaru@bitdefender.com>
Acked-by: Tamas K Lengyel <tlengyel@novetta.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>

---
Changes since V4:
 - Added XEN_DOMCTL_MONITOR_EVENT_GUEST_REQUEST to the capabilities
   in monitor.c (Tamas' patch has hit staging since V4).
---
 tools/libxc/include/xenctrl.h   |    2 ++
 tools/libxc/xc_monitor.c        |   15 +++++++++++++++
 xen/arch/x86/hvm/event.c        |   16 ++++++++++++++++
 xen/arch/x86/hvm/hvm.c          |    8 +++++++-
 xen/arch/x86/monitor.c          |   19 ++++++++++++++++++-
 xen/include/asm-x86/domain.h    |   16 +++++++++-------
 xen/include/asm-x86/hvm/event.h |    1 +
 xen/include/public/domctl.h     |    6 ++++++
 xen/include/public/hvm/hvm_op.h |    2 ++
 xen/include/public/vm_event.h   |    2 ++
 10 files changed, 78 insertions(+), 9 deletions(-)

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index 0bbae2a..ce9029c 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2378,6 +2378,8 @@ int xc_monitor_mov_to_msr(xc_interface *xch, domid_t domain_id, bool enable,
 int xc_monitor_singlestep(xc_interface *xch, domid_t domain_id, bool enable);
 int xc_monitor_software_breakpoint(xc_interface *xch, domid_t domain_id,
                                    bool enable);
+int xc_monitor_guest_request(xc_interface *xch, domid_t domain_id,
+                             bool enable, bool sync);
 
 /***
  * Memory sharing operations.
diff --git a/tools/libxc/xc_monitor.c b/tools/libxc/xc_monitor.c
index b64bce3..d5f87da 100644
--- a/tools/libxc/xc_monitor.c
+++ b/tools/libxc/xc_monitor.c
@@ -129,3 +129,18 @@ int xc_monitor_singlestep(xc_interface *xch, domid_t domain_id,
 
     return do_domctl(xch, &domctl);
 }
+
+int xc_monitor_guest_request(xc_interface *xch, domid_t domain_id, bool enable,
+                             bool sync)
+{
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_monitor_op;
+    domctl.domain = domain_id;
+    domctl.u.monitor_op.op = enable ? XEN_DOMCTL_MONITOR_OP_ENABLE
+                                    : XEN_DOMCTL_MONITOR_OP_DISABLE;
+    domctl.u.monitor_op.event = XEN_DOMCTL_MONITOR_EVENT_GUEST_REQUEST;
+    domctl.u.monitor_op.u.guest_request.sync = sync;
+
+    return do_domctl(xch, &domctl);
+}
diff --git a/xen/arch/x86/hvm/event.c b/xen/arch/x86/hvm/event.c
index 5341937..17638ea 100644
--- a/xen/arch/x86/hvm/event.c
+++ b/xen/arch/x86/hvm/event.c
@@ -126,6 +126,22 @@ void hvm_event_msr(unsigned int msr, uint64_t value)
         hvm_event_traps(1, &req);
 }
 
+void hvm_event_guest_request(void)
+{
+    struct vcpu *curr = current;
+    struct arch_domain *currad = &curr->domain->arch;
+
+    if ( currad->monitor.guest_request_enabled )
+    {
+        vm_event_request_t req = {
+            .reason = VM_EVENT_REASON_GUEST_REQUEST,
+            .vcpu_id = curr->vcpu_id,
+        };
+
+        hvm_event_traps(currad->monitor.guest_request_sync, &req);
+    }
+}
+
 int hvm_event_int3(unsigned long gla)
 {
     int rc = 0;
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 545aa91..18d9621 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -6001,7 +6001,6 @@ static int hvmop_get_param(
 #define HVMOP_op_mask 0xff
 
 long do_hvm_op(unsigned long op, XEN_GUEST_HANDLE_PARAM(void) arg)
-
 {
     unsigned long start_iter, mask;
     long rc = 0;
@@ -6415,6 +6414,13 @@ long do_hvm_op(unsigned long op, XEN_GUEST_HANDLE_PARAM(void) arg)
         break;
     }
 
+    case HVMOP_guest_request_vm_event:
+        if ( guest_handle_is_null(arg) )
+            hvm_event_guest_request();
+        else
+            rc = -EINVAL;
+        break;
+
     default:
     {
         gdprintk(XENLOG_DEBUG, "Bad HVM op %ld.\n", op);
diff --git a/xen/arch/x86/monitor.c b/xen/arch/x86/monitor.c
index 0da855e..d35907b 100644
--- a/xen/arch/x86/monitor.c
+++ b/xen/arch/x86/monitor.c
@@ -55,7 +55,8 @@ static inline uint32_t get_capabilities(struct domain *d)
 
     capabilities = (1 << XEN_DOMCTL_MONITOR_EVENT_WRITE_CTRLREG) |
                    (1 << XEN_DOMCTL_MONITOR_EVENT_MOV_TO_MSR) |
-                   (1 << XEN_DOMCTL_MONITOR_EVENT_SOFTWARE_BREAKPOINT);
+                   (1 << XEN_DOMCTL_MONITOR_EVENT_SOFTWARE_BREAKPOINT) |
+                   (1 << XEN_DOMCTL_MONITOR_EVENT_GUEST_REQUEST);
 
     /* Since we know this is on VMX, we can just call the hvm func */
     if ( hvm_is_singlestep_supported() )
@@ -184,6 +185,22 @@ int monitor_domctl(struct domain *d, struct xen_domctl_monitor_op *mop)
         break;
     }
 
+    case XEN_DOMCTL_MONITOR_EVENT_GUEST_REQUEST:
+    {
+        bool_t status = ad->monitor.guest_request_enabled;
+
+        rc = status_check(mop, status);
+        if ( rc )
+            return rc;
+
+        ad->monitor.guest_request_sync = mop->u.guest_request.sync;
+
+        domain_pause(d);
+        ad->monitor.guest_request_enabled = !status;
+        domain_unpause(d);
+        break;
+    }
+
     default:
         return -EOPNOTSUPP;
 
diff --git a/xen/include/asm-x86/domain.h b/xen/include/asm-x86/domain.h
index c5ad1cb..9fbbdd9 100644
--- a/xen/include/asm-x86/domain.h
+++ b/xen/include/asm-x86/domain.h
@@ -347,13 +347,15 @@ struct arch_domain
 
     /* Monitor options */
     struct {
-        uint16_t write_ctrlreg_enabled       : 4;
-        uint16_t write_ctrlreg_sync          : 4;
-        uint16_t write_ctrlreg_onchangeonly  : 4;
-        uint16_t mov_to_msr_enabled          : 1;
-        uint16_t mov_to_msr_extended         : 1;
-        uint16_t singlestep_enabled          : 1;
-        uint16_t software_breakpoint_enabled : 1;
+        unsigned int write_ctrlreg_enabled       : 4;
+        unsigned int write_ctrlreg_sync          : 4;
+        unsigned int write_ctrlreg_onchangeonly  : 4;
+        unsigned int mov_to_msr_enabled          : 1;
+        unsigned int mov_to_msr_extended         : 1;
+        unsigned int singlestep_enabled          : 1;
+        unsigned int software_breakpoint_enabled : 1;
+        unsigned int guest_request_enabled       : 1;
+        unsigned int guest_request_sync          : 1;
     } monitor;
 
     /* Mem_access emulation control */
diff --git a/xen/include/asm-x86/hvm/event.h b/xen/include/asm-x86/hvm/event.h
index 819ef96..ab5abd0 100644
--- a/xen/include/asm-x86/hvm/event.h
+++ b/xen/include/asm-x86/hvm/event.h
@@ -26,6 +26,7 @@ void hvm_event_msr(unsigned int msr, uint64_t value);
 /* Called for current VCPU: returns -1 if no listener */
 int hvm_event_int3(unsigned long gla);
 int hvm_event_single_step(unsigned long gla);
+void hvm_event_guest_request(void);
 
 #endif /* __ASM_X86_HVM_EVENT_H__ */
 
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 8b1d6b4..631935a 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -1009,6 +1009,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_domctl_psr_cmt_op_t);
 #define XEN_DOMCTL_MONITOR_EVENT_MOV_TO_MSR            1
 #define XEN_DOMCTL_MONITOR_EVENT_SINGLESTEP            2
 #define XEN_DOMCTL_MONITOR_EVENT_SOFTWARE_BREAKPOINT   3
+#define XEN_DOMCTL_MONITOR_EVENT_GUEST_REQUEST         4
 
 struct xen_domctl_monitor_op {
     uint32_t op; /* XEN_DOMCTL_MONITOR_OP_* */
@@ -1039,6 +1040,11 @@ struct xen_domctl_monitor_op {
             /* Enable the capture of an extended set of MSRs */
             uint8_t extended_capture;
         } mov_to_msr;
+
+        struct {
+            /* Pause vCPU until response */
+            uint8_t sync;
+        } guest_request;
     } u;
 };
 typedef struct xen_domctl_monitor_op xen_domctl_monitor_op_t;
diff --git a/xen/include/public/hvm/hvm_op.h b/xen/include/public/hvm/hvm_op.h
index 9b84e84..d053db9 100644
--- a/xen/include/public/hvm/hvm_op.h
+++ b/xen/include/public/hvm/hvm_op.h
@@ -396,6 +396,8 @@ DEFINE_XEN_GUEST_HANDLE(xen_hvm_evtchn_upcall_vector_t);
 
 #endif /* defined(__i386__) || defined(__x86_64__) */
 
+#define HVMOP_guest_request_vm_event 24
+
 #endif /* __XEN_PUBLIC_HVM_HVM_OP_H__ */
 
 /*
diff --git a/xen/include/public/vm_event.h b/xen/include/public/vm_event.h
index 4d89c38..f889139 100644
--- a/xen/include/public/vm_event.h
+++ b/xen/include/public/vm_event.h
@@ -95,6 +95,8 @@
 #define VM_EVENT_REASON_SOFTWARE_BREAKPOINT     6
 /* Single-step (e.g. MTF) */
 #define VM_EVENT_REASON_SINGLESTEP              7
+/* An event has been requested via HVMOP_guest_request_vm_event. */
+#define VM_EVENT_REASON_GUEST_REQUEST           8
 
 /* Supported values for the vm_event_write_ctrlreg index. */
 #define VM_EVENT_X86_CR0    0
-- 
1.7.9.5

[PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Razvan Cojocaru]

Deny register writes if a vm_client subscribed to mov_to_msr or
control register write events forbids them. Currently supported for
MSR, CR0, CR3 and CR4 events.

Signed-off-by: Razvan Cojocaru <rcojocaru@bitdefender.com>
Acked-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Tamas K Lengyel <tlengyel@novetta.com>

---
Changes since V4:
 - Rebased the patch to take into account Tamas' "x86/vm_event:
   toggle singlestep from vm_event response".
 - Moved the management of heap-allocated vm_event-related
   domain data to the specific x86 vm_event domain init / cleanup
   helpers.
---
 xen/arch/x86/domain.c             |    2 +
 xen/arch/x86/hvm/emulate.c        |    8 +--
 xen/arch/x86/hvm/event.c          |    5 +-
 xen/arch/x86/hvm/hvm.c            |  118 ++++++++++++++++++++++++++++++++-----
 xen/arch/x86/hvm/svm/nestedsvm.c  |   14 ++---
 xen/arch/x86/hvm/svm/svm.c        |    2 +-
 xen/arch/x86/hvm/vmx/vmx.c        |   15 +++--
 xen/arch/x86/hvm/vmx/vvmx.c       |   18 +++---
 xen/arch/x86/vm_event.c           |   43 ++++++++++++++
 xen/common/vm_event.c             |    4 ++
 xen/include/asm-arm/vm_event.h    |    7 +++
 xen/include/asm-x86/domain.h      |   18 +++++-
 xen/include/asm-x86/hvm/event.h   |    9 ++-
 xen/include/asm-x86/hvm/support.h |    9 +--
 xen/include/asm-x86/vm_event.h    |    3 +
 xen/include/public/vm_event.h     |    5 ++
 16 files changed, 234 insertions(+), 46 deletions(-)

diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 6596408..b940b04 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -667,6 +667,8 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags,
 
 void arch_domain_destroy(struct domain *d)
 {
+    xfree(d->arch.event_write_data);
+
     if ( has_hvm_container_domain(d) )
         hvm_domain_destroy(d);
 
diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
index 30def25..9aae0df 100644
--- a/xen/arch/x86/hvm/emulate.c
+++ b/xen/arch/x86/hvm/emulate.c
@@ -1427,14 +1427,14 @@ static int hvmemul_write_cr(
     switch ( reg )
     {
     case 0:
-        return hvm_set_cr0(val);
+        return hvm_set_cr0(val, 1);
     case 2:
         current->arch.hvm_vcpu.guest_cr[2] = val;
         return X86EMUL_OKAY;
     case 3:
-        return hvm_set_cr3(val);
+        return hvm_set_cr3(val, 1);
     case 4:
-        return hvm_set_cr4(val);
+        return hvm_set_cr4(val, 1);
     default:
         break;
     }
@@ -1455,7 +1455,7 @@ static int hvmemul_write_msr(
     uint64_t val,
     struct x86_emulate_ctxt *ctxt)
 {
-    return hvm_msr_write_intercept(reg, val);
+    return hvm_msr_write_intercept(reg, val, 1);
 }
 
 static int hvmemul_wbinvd(
diff --git a/xen/arch/x86/hvm/event.c b/xen/arch/x86/hvm/event.c
index 17638ea..042e583 100644
--- a/xen/arch/x86/hvm/event.c
+++ b/xen/arch/x86/hvm/event.c
@@ -90,7 +90,7 @@ static int hvm_event_traps(uint8_t sync, vm_event_request_t *req)
     return 1;
 }
 
-void hvm_event_cr(unsigned int index, unsigned long value, unsigned long old)
+bool_t hvm_event_cr(unsigned int index, unsigned long value, unsigned long old)
 {
     struct arch_domain *currad = &current->domain->arch;
     unsigned int ctrlreg_bitmask = monitor_ctrlreg_bitmask(index);
@@ -109,7 +109,10 @@ void hvm_event_cr(unsigned int index, unsigned long value, unsigned long old)
 
         hvm_event_traps(currad->monitor.write_ctrlreg_sync & ctrlreg_bitmask,
                         &req);
+        return 1;
     }
+
+    return 0;
 }
 
 void hvm_event_msr(unsigned int msr, uint64_t value)
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 18d9621..296bbb0 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -52,6 +52,7 @@
 #include <asm/traps.h>
 #include <asm/mc146818rtc.h>
 #include <asm/mce.h>
+#include <asm/monitor.h>
 #include <asm/hvm/hvm.h>
 #include <asm/hvm/vpt.h>
 #include <asm/hvm/support.h>
@@ -519,6 +520,35 @@ void hvm_do_resume(struct vcpu *v)
         break;
     }
 
+    if ( unlikely(d->arch.event_write_data) )
+    {
+        struct monitor_write_data *w = &d->arch.event_write_data[v->vcpu_id];
+
+        if ( w->do_write.msr )
+        {
+            hvm_msr_write_intercept(w->msr, w->value, 0);
+            w->do_write.msr = 0;
+        }
+
+        if ( w->do_write.cr0 )
+        {
+            hvm_set_cr0(w->cr0, 0);
+            w->do_write.cr0 = 0;
+        }
+
+        if ( w->do_write.cr4 )
+        {
+            hvm_set_cr4(w->cr4, 0);
+            w->do_write.cr4 = 0;
+        }
+
+        if ( w->do_write.cr3 )
+        {
+            hvm_set_cr3(w->cr3, 0);
+            w->do_write.cr3 = 0;
+        }
+    }
+
     /* Inject pending hw/sw trap */
     if ( v->arch.hvm_vcpu.inject_trap.vector != -1 ) 
     {
@@ -3125,13 +3155,13 @@ int hvm_mov_to_cr(unsigned int cr, unsigned int gpr)
     switch ( cr )
     {
     case 0:
-        return hvm_set_cr0(val);
+        return hvm_set_cr0(val, 1);
 
     case 3:
-        return hvm_set_cr3(val);
+        return hvm_set_cr3(val, 1);
 
     case 4:
-        return hvm_set_cr4(val);
+        return hvm_set_cr4(val, 1);
 
     case 8:
         vlapic_set_reg(vcpu_vlapic(curr), APIC_TASKPRI, ((val & 0x0f) << 4));
@@ -3228,12 +3258,13 @@ static void hvm_update_cr(struct vcpu *v, unsigned int cr, unsigned long value)
     hvm_update_guest_cr(v, cr);
 }
 
-int hvm_set_cr0(unsigned long value)
+int hvm_set_cr0(unsigned long value, bool_t may_defer)
 {
     struct vcpu *v = current;
     struct domain *d = v->domain;
     unsigned long gfn, old_value = v->arch.hvm_vcpu.guest_cr[0];
     struct page_info *page;
+    struct arch_domain *currad = &v->domain->arch;
 
     HVM_DBG_LOG(DBG_LEVEL_VMMU, "Update CR0 value = %lx", value);
 
@@ -3263,6 +3294,22 @@ int hvm_set_cr0(unsigned long value)
         goto gpf;
     }
 
+    if ( may_defer && unlikely(currad->monitor.write_ctrlreg_enabled &
+                               monitor_ctrlreg_bitmask(VM_EVENT_X86_CR0)) &&
+         value != old_value )
+    {
+        ASSERT(currad->event_write_data != NULL);
+
+        if ( hvm_event_crX(CR0, value, old_value) )
+        {
+            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            currad->event_write_data[v->vcpu_id].do_write.cr0 = 1;
+            currad->event_write_data[v->vcpu_id].cr0 = value;
+
+            return X86EMUL_OKAY;
+        }
+    }
+
     if ( (value & X86_CR0_PG) && !(old_value & X86_CR0_PG) )
     {
         if ( v->arch.hvm_vcpu.guest_efer & EFER_LME )
@@ -3329,7 +3376,6 @@ int hvm_set_cr0(unsigned long value)
         hvm_funcs.handle_cd(v, value);
 
     hvm_update_cr(v, 0, value);
-    hvm_event_crX(CR0, value, old_value);
 
     if ( (value ^ old_value) & X86_CR0_PG ) {
         if ( !nestedhvm_vmswitch_in_progress(v) && nestedhvm_vcpu_in_guestmode(v) )
@@ -3345,11 +3391,28 @@ int hvm_set_cr0(unsigned long value)
     return X86EMUL_EXCEPTION;
 }
 
-int hvm_set_cr3(unsigned long value)
+int hvm_set_cr3(unsigned long value, bool_t may_defer)
 {
     struct vcpu *v = current;
     struct page_info *page;
-    unsigned long old;
+    unsigned long old = v->arch.hvm_vcpu.guest_cr[3];
+    struct arch_domain *currad = &v->domain->arch;
+
+    if ( may_defer && unlikely(currad->monitor.write_ctrlreg_enabled &
+                               monitor_ctrlreg_bitmask(VM_EVENT_X86_CR3)) &&
+         value != old )
+    {
+        ASSERT(currad->event_write_data != NULL);
+
+        if ( hvm_event_crX(CR3, value, old) )
+        {
+            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            currad->event_write_data[v->vcpu_id].do_write.cr3 = 1;
+            currad->event_write_data[v->vcpu_id].cr3 = value;
+
+            return X86EMUL_OKAY;
+        }
+    }
 
     if ( hvm_paging_enabled(v) && !paging_mode_hap(v->domain) &&
          (value != v->arch.hvm_vcpu.guest_cr[3]) )
@@ -3367,10 +3430,8 @@ int hvm_set_cr3(unsigned long value)
         HVM_DBG_LOG(DBG_LEVEL_VMMU, "Update CR3 value = %lx", value);
     }
 
-    old=v->arch.hvm_vcpu.guest_cr[3];
     v->arch.hvm_vcpu.guest_cr[3] = value;
     paging_update_cr3(v);
-    hvm_event_crX(CR3, value, old);
     return X86EMUL_OKAY;
 
  bad_cr3:
@@ -3379,10 +3440,11 @@ int hvm_set_cr3(unsigned long value)
     return X86EMUL_UNHANDLEABLE;
 }
 
-int hvm_set_cr4(unsigned long value)
+int hvm_set_cr4(unsigned long value, bool_t may_defer)
 {
     struct vcpu *v = current;
     unsigned long old_cr;
+    struct arch_domain *currad = &v->domain->arch;
 
     if ( value & hvm_cr4_guest_reserved_bits(v, 0) )
     {
@@ -3410,8 +3472,23 @@ int hvm_set_cr4(unsigned long value)
         goto gpf;
     }
 
+    if ( may_defer && unlikely(currad->monitor.write_ctrlreg_enabled &
+                               monitor_ctrlreg_bitmask(VM_EVENT_X86_CR4)) &&
+         value != old_cr )
+    {
+        ASSERT(currad->event_write_data != NULL);
+
+        if ( hvm_event_crX(CR4, value, old_cr) )
+        {
+            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            currad->event_write_data[v->vcpu_id].do_write.cr4 = 1;
+            currad->event_write_data[v->vcpu_id].cr4 = value;
+
+            return X86EMUL_OKAY;
+        }
+    }
+
     hvm_update_cr(v, 4, value);
-    hvm_event_crX(CR4, value, old_cr);
 
     /*
      * Modifying CR4.{PSE,PAE,PGE,SMEP}, or clearing CR4.PCIDE
@@ -3875,7 +3952,7 @@ void hvm_task_switch(
         goto out;
 
 
-    if ( hvm_set_cr3(tss.cr3) )
+    if ( hvm_set_cr3(tss.cr3, 1) )
         goto out;
 
     regs->eip    = tss.eip;
@@ -4577,12 +4654,14 @@ int hvm_msr_read_intercept(unsigned int msr, uint64_t *msr_content)
     goto out;
 }
 
-int hvm_msr_write_intercept(unsigned int msr, uint64_t msr_content)
+int hvm_msr_write_intercept(unsigned int msr, uint64_t msr_content,
+                            bool_t may_defer)
 {
     struct vcpu *v = current;
     bool_t mtrr;
     unsigned int edx, index;
     int ret = X86EMUL_OKAY;
+    struct arch_domain *currad = &current->domain->arch;
 
     HVMTRACE_3D(MSR_WRITE, msr,
                (uint32_t)msr_content, (uint32_t)(msr_content >> 32));
@@ -4590,7 +4669,18 @@ int hvm_msr_write_intercept(unsigned int msr, uint64_t msr_content)
     hvm_cpuid(1, NULL, NULL, NULL, &edx);
     mtrr = !!(edx & cpufeat_mask(X86_FEATURE_MTRR));
 
-    hvm_event_msr(msr, msr_content);
+    if ( may_defer && unlikely(currad->monitor.mov_to_msr_enabled) )
+    {
+        ASSERT(currad->event_write_data != NULL);
+
+        /* The actual write will occur in hvm_do_resume() (if permitted). */
+        currad->event_write_data[v->vcpu_id].do_write.msr = 1;
+        currad->event_write_data[v->vcpu_id].msr = msr;
+        currad->event_write_data[v->vcpu_id].value = msr_content;
+
+        hvm_event_msr(msr, msr_content);
+        return X86EMUL_OKAY;
+    }
 
     switch ( msr )
     {
diff --git a/xen/arch/x86/hvm/svm/nestedsvm.c b/xen/arch/x86/hvm/svm/nestedsvm.c
index 2653bc1..f22b7d1 100644
--- a/xen/arch/x86/hvm/svm/nestedsvm.c
+++ b/xen/arch/x86/hvm/svm/nestedsvm.c
@@ -274,7 +274,7 @@ static int nsvm_vcpu_hostrestore(struct vcpu *v, struct cpu_user_regs *regs)
 
     /* CR4 */
     v->arch.hvm_vcpu.guest_cr[4] = n1vmcb->_cr4;
-    rc = hvm_set_cr4(n1vmcb->_cr4);
+    rc = hvm_set_cr4(n1vmcb->_cr4, 1);
     if (rc != X86EMUL_OKAY)
         gdprintk(XENLOG_ERR, "hvm_set_cr4 failed, rc: %u\n", rc);
 
@@ -283,7 +283,7 @@ static int nsvm_vcpu_hostrestore(struct vcpu *v, struct cpu_user_regs *regs)
         svm->ns_cr0, v->arch.hvm_vcpu.guest_cr[0]);
     v->arch.hvm_vcpu.guest_cr[0] = n1vmcb->_cr0 | X86_CR0_PE;
     n1vmcb->rflags &= ~X86_EFLAGS_VM;
-    rc = hvm_set_cr0(n1vmcb->_cr0 | X86_CR0_PE);
+    rc = hvm_set_cr0(n1vmcb->_cr0 | X86_CR0_PE, 1);
     if (rc != X86EMUL_OKAY)
         gdprintk(XENLOG_ERR, "hvm_set_cr0 failed, rc: %u\n", rc);
     svm->ns_cr0 = v->arch.hvm_vcpu.guest_cr[0];
@@ -309,7 +309,7 @@ static int nsvm_vcpu_hostrestore(struct vcpu *v, struct cpu_user_regs *regs)
         v->arch.guest_table = pagetable_null();
         /* hvm_set_cr3() below sets v->arch.hvm_vcpu.guest_cr[3] for us. */
     }
-    rc = hvm_set_cr3(n1vmcb->_cr3);
+    rc = hvm_set_cr3(n1vmcb->_cr3, 1);
     if (rc != X86EMUL_OKAY)
         gdprintk(XENLOG_ERR, "hvm_set_cr3 failed, rc: %u\n", rc);
 
@@ -534,7 +534,7 @@ static int nsvm_vmcb_prepare4vmrun(struct vcpu *v, struct cpu_user_regs *regs)
 
     /* CR4 */
     v->arch.hvm_vcpu.guest_cr[4] = ns_vmcb->_cr4;
-    rc = hvm_set_cr4(ns_vmcb->_cr4);
+    rc = hvm_set_cr4(ns_vmcb->_cr4, 1);
     if (rc != X86EMUL_OKAY)
         gdprintk(XENLOG_ERR, "hvm_set_cr4 failed, rc: %u\n", rc);
 
@@ -542,7 +542,7 @@ static int nsvm_vmcb_prepare4vmrun(struct vcpu *v, struct cpu_user_regs *regs)
     svm->ns_cr0 = v->arch.hvm_vcpu.guest_cr[0];
     cr0 = nestedsvm_fpu_vmentry(svm->ns_cr0, ns_vmcb, n1vmcb, n2vmcb);
     v->arch.hvm_vcpu.guest_cr[0] = ns_vmcb->_cr0;
-    rc = hvm_set_cr0(cr0);
+    rc = hvm_set_cr0(cr0, 1);
     if (rc != X86EMUL_OKAY)
         gdprintk(XENLOG_ERR, "hvm_set_cr0 failed, rc: %u\n", rc);
 
@@ -558,7 +558,7 @@ static int nsvm_vmcb_prepare4vmrun(struct vcpu *v, struct cpu_user_regs *regs)
         nestedsvm_vmcb_set_nestedp2m(v, ns_vmcb, n2vmcb);
 
         /* hvm_set_cr3() below sets v->arch.hvm_vcpu.guest_cr[3] for us. */
-        rc = hvm_set_cr3(ns_vmcb->_cr3);
+        rc = hvm_set_cr3(ns_vmcb->_cr3, 1);
         if (rc != X86EMUL_OKAY)
             gdprintk(XENLOG_ERR, "hvm_set_cr3 failed, rc: %u\n", rc);
     } else if (paging_mode_hap(v->domain)) {
@@ -570,7 +570,7 @@ static int nsvm_vmcb_prepare4vmrun(struct vcpu *v, struct cpu_user_regs *regs)
          * we assume it intercepts page faults.
          */
         /* hvm_set_cr3() below sets v->arch.hvm_vcpu.guest_cr[3] for us. */
-        rc = hvm_set_cr3(ns_vmcb->_cr3);
+        rc = hvm_set_cr3(ns_vmcb->_cr3, 1);
         if (rc != X86EMUL_OKAY)
             gdprintk(XENLOG_ERR, "hvm_set_cr3 failed, rc: %u\n", rc);
     } else {
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 70de49e..b8bba71 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -1949,7 +1949,7 @@ static void svm_do_msr_access(struct cpu_user_regs *regs)
         if ( (inst_len = __get_instruction_length(v, INSTR_WRMSR)) == 0 )
             return;
         msr_content = ((uint64_t)regs->edx << 32) | (uint32_t)regs->eax;
-        rc = hvm_msr_write_intercept(regs->ecx, msr_content);
+        rc = hvm_msr_write_intercept(regs->ecx, msr_content, 1);
     }
 
     if ( rc == X86EMUL_OKAY )
diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index bc3212f..d3183a8 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -2016,9 +2016,16 @@ static int vmx_cr_access(unsigned long exit_qualification)
     }
     case VMX_CONTROL_REG_ACCESS_TYPE_CLTS: {
         unsigned long old = curr->arch.hvm_vcpu.guest_cr[0];
-        curr->arch.hvm_vcpu.guest_cr[0] &= ~X86_CR0_TS;
+        unsigned long value = old & ~X86_CR0_TS;
+
+        /*
+         * Special case unlikely to be interesting to a
+         * VM_EVENT_FLAG_DENY-capable application, so the hvm_event_crX()
+         * return value is ignored for now.
+         */
+        hvm_event_crX(CR0, value, old);
+        curr->arch.hvm_vcpu.guest_cr[0] = value;
         vmx_update_guest_cr(curr, 0);
-        hvm_event_crX(CR0, curr->arch.hvm_vcpu.guest_cr[0], old);
         HVMTRACE_0D(CLTS);
         break;
     }
@@ -2030,7 +2037,7 @@ static int vmx_cr_access(unsigned long exit_qualification)
                 (VMX_CONTROL_REG_ACCESS_DATA(exit_qualification) &
                  (X86_CR0_PE|X86_CR0_MP|X86_CR0_EM|X86_CR0_TS));
         HVMTRACE_LONG_1D(LMSW, value);
-        return hvm_set_cr0(value);
+        return hvm_set_cr0(value, 1);
     }
     default:
         BUG();
@@ -3053,7 +3060,7 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
     {
         uint64_t msr_content;
         msr_content = ((uint64_t)regs->edx << 32) | (uint32_t)regs->eax;
-        if ( hvm_msr_write_intercept(regs->ecx, msr_content) == X86EMUL_OKAY )
+        if ( hvm_msr_write_intercept(regs->ecx, msr_content, 1) == X86EMUL_OKAY )
             update_guest_eip(); /* Safe: WRMSR */
         break;
     }
diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c
index 72dd9c8..555fdfa 100644
--- a/xen/arch/x86/hvm/vmx/vvmx.c
+++ b/xen/arch/x86/hvm/vmx/vvmx.c
@@ -1034,15 +1034,16 @@ static void load_shadow_guest_state(struct vcpu *v)
 
     nvcpu->guest_cr[0] = __get_vvmcs(vvmcs, CR0_READ_SHADOW);
     nvcpu->guest_cr[4] = __get_vvmcs(vvmcs, CR4_READ_SHADOW);
-    hvm_set_cr0(__get_vvmcs(vvmcs, GUEST_CR0));
-    hvm_set_cr4(__get_vvmcs(vvmcs, GUEST_CR4));
-    hvm_set_cr3(__get_vvmcs(vvmcs, GUEST_CR3));
+    hvm_set_cr0(__get_vvmcs(vvmcs, GUEST_CR0), 1);
+    hvm_set_cr4(__get_vvmcs(vvmcs, GUEST_CR4), 1);
+    hvm_set_cr3(__get_vvmcs(vvmcs, GUEST_CR3), 1);
 
     control = __get_vvmcs(vvmcs, VM_ENTRY_CONTROLS);
     if ( control & VM_ENTRY_LOAD_GUEST_PAT )
         hvm_set_guest_pat(v, __get_vvmcs(vvmcs, GUEST_PAT));
     if ( control & VM_ENTRY_LOAD_PERF_GLOBAL_CTRL )
-        hvm_msr_write_intercept(MSR_CORE_PERF_GLOBAL_CTRL, __get_vvmcs(vvmcs, GUEST_PERF_GLOBAL_CTRL));
+        hvm_msr_write_intercept(MSR_CORE_PERF_GLOBAL_CTRL,
+                                __get_vvmcs(vvmcs, GUEST_PERF_GLOBAL_CTRL), 0);
 
     hvm_funcs.set_tsc_offset(v, v->arch.hvm_vcpu.cache_tsc_offset, 0);
 
@@ -1235,15 +1236,16 @@ static void load_vvmcs_host_state(struct vcpu *v)
         __vmwrite(vmcs_h2g_field[i].guest_field, r);
     }
 
-    hvm_set_cr0(__get_vvmcs(vvmcs, HOST_CR0));
-    hvm_set_cr4(__get_vvmcs(vvmcs, HOST_CR4));
-    hvm_set_cr3(__get_vvmcs(vvmcs, HOST_CR3));
+    hvm_set_cr0(__get_vvmcs(vvmcs, HOST_CR0), 1);
+    hvm_set_cr4(__get_vvmcs(vvmcs, HOST_CR4), 1);
+    hvm_set_cr3(__get_vvmcs(vvmcs, HOST_CR3), 1);
 
     control = __get_vvmcs(vvmcs, VM_EXIT_CONTROLS);
     if ( control & VM_EXIT_LOAD_HOST_PAT )
         hvm_set_guest_pat(v, __get_vvmcs(vvmcs, HOST_PAT));
     if ( control & VM_EXIT_LOAD_PERF_GLOBAL_CTRL )
-        hvm_msr_write_intercept(MSR_CORE_PERF_GLOBAL_CTRL, __get_vvmcs(vvmcs, HOST_PERF_GLOBAL_CTRL));
+        hvm_msr_write_intercept(MSR_CORE_PERF_GLOBAL_CTRL,
+                                __get_vvmcs(vvmcs, HOST_PERF_GLOBAL_CTRL), 1);
 
     hvm_funcs.set_tsc_offset(v, v->arch.hvm_vcpu.cache_tsc_offset, 0);
 
diff --git a/xen/arch/x86/vm_event.c b/xen/arch/x86/vm_event.c
index 0515161..5eb1f8d 100644
--- a/xen/arch/x86/vm_event.c
+++ b/xen/arch/x86/vm_event.c
@@ -22,11 +22,19 @@
 
 #include <xen/sched.h>
 #include <asm/hvm/hvm.h>
+#include <asm/vm_event.h>
 
 int vm_event_init_domain(struct domain *d)
 {
     struct vcpu *v;
 
+    if ( !d->arch.event_write_data )
+        d->arch.event_write_data = xzalloc_array(struct monitor_write_data,
+                                                 d->max_vcpus);
+
+    if ( !d->arch.event_write_data )
+        return -ENOMEM;
+
     for_each_vcpu( d, v )
     {
         if ( v->arch.vm_event.emul_read_data )
@@ -46,6 +54,9 @@ void vm_event_cleanup_domain(struct domain *d)
 {
     struct vcpu *v;
 
+    xfree(d->arch.event_write_data);
+    d->arch.event_write_data = NULL;
+
     for_each_vcpu( d, v )
     {
         xfree(v->arch.vm_event.emul_read_data);
@@ -61,6 +72,38 @@ void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v)
     hvm_toggle_singlestep(v);
 }
 
+void vm_event_register_write_resume(struct vcpu *v, vm_event_response_t *rsp)
+{
+    if ( rsp->flags & VM_EVENT_FLAG_DENY )
+    {
+        struct monitor_write_data *w =
+            &v->domain->arch.event_write_data[v->vcpu_id];
+
+        ASSERT(v->domain->arch.event_write_data != NULL);
+
+        switch ( rsp->reason )
+        {
+        case VM_EVENT_REASON_MOV_TO_MSR:
+            w->do_write.msr = 0;
+            break;
+        case VM_EVENT_REASON_WRITE_CTRLREG:
+            switch ( rsp->u.write_ctrlreg.index )
+            {
+            case VM_EVENT_X86_CR0:
+                w->do_write.cr0 = 0;
+                break;
+            case VM_EVENT_X86_CR3:
+                w->do_write.cr3 = 0;
+                break;
+            case VM_EVENT_X86_CR4:
+                w->do_write.cr4 = 0;
+                break;
+            }
+            break;
+        }
+    }
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/common/vm_event.c b/xen/common/vm_event.c
index 0007d70..4c6bf98 100644
--- a/xen/common/vm_event.c
+++ b/xen/common/vm_event.c
@@ -393,6 +393,10 @@ void vm_event_resume(struct domain *d, struct vm_event_domain *ved)
          */
         switch ( rsp.reason )
         {
+        case VM_EVENT_REASON_MOV_TO_MSR:
+        case VM_EVENT_REASON_WRITE_CTRLREG:
+            vm_event_register_write_resume(v, &rsp);
+            break;
 
 #ifdef HAS_MEM_ACCESS
         case VM_EVENT_REASON_MEM_ACCESS:
diff --git a/xen/include/asm-arm/vm_event.h b/xen/include/asm-arm/vm_event.h
index 20469a8..0833a65 100644
--- a/xen/include/asm-arm/vm_event.h
+++ b/xen/include/asm-arm/vm_event.h
@@ -21,6 +21,7 @@
 #define __ASM_ARM_VM_EVENT_H__
 
 #include <xen/sched.h>
+#include <xen/vm_event.h>
 
 static inline
 int vm_event_init_domain(struct domain *d)
@@ -41,4 +42,10 @@ void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v)
     /* Not supported on ARM. */
 }
 
+static inline
+void vm_event_register_write_resume(struct vcpu *v, vm_event_response_t *rsp)
+{
+    /* Not supported on ARM. */
+}
+
 #endif /* __ASM_ARM_VM_EVENT_H__ */
diff --git a/xen/include/asm-x86/domain.h b/xen/include/asm-x86/domain.h
index 9fbbdd9..7a9e96f 100644
--- a/xen/include/asm-x86/domain.h
+++ b/xen/include/asm-x86/domain.h
@@ -247,6 +247,21 @@ struct pv_domain
     struct mapcache_domain mapcache;
 };
 
+struct monitor_write_data {
+    struct {
+        unsigned int msr : 1;
+        unsigned int cr0 : 1;
+        unsigned int cr3 : 1;
+        unsigned int cr4 : 1;
+    } do_write;
+
+    uint32_t msr;
+    uint64_t value;
+    uint64_t cr0;
+    uint64_t cr3;
+    uint64_t cr4;
+};
+
 struct arch_domain
 {
     struct page_info *perdomain_l3_pg;
@@ -360,6 +375,8 @@ struct arch_domain
 
     /* Mem_access emulation control */
     bool_t mem_access_emulate_enabled;
+
+    struct monitor_write_data *event_write_data;
 } __cacheline_aligned;
 
 #define has_arch_pdevs(d)    (!list_empty(&(d)->arch.pdev_list))
@@ -516,7 +533,6 @@ struct arch_vcpu
         unsigned long eip;
         struct vm_event_emul_read_data *emul_read_data;
     } vm_event;
-
 };
 
 smap_check_policy_t smap_policy_change(struct vcpu *v,
diff --git a/xen/include/asm-x86/hvm/event.h b/xen/include/asm-x86/hvm/event.h
index ab5abd0..c082c20 100644
--- a/xen/include/asm-x86/hvm/event.h
+++ b/xen/include/asm-x86/hvm/event.h
@@ -18,8 +18,13 @@
 #ifndef __ASM_X86_HVM_EVENT_H__
 #define __ASM_X86_HVM_EVENT_H__
 
-/* Called for current VCPU on crX/MSR changes by guest */
-void hvm_event_cr(unsigned int index, unsigned long value, unsigned long old);
+/*
+ * Called for current VCPU on crX/MSR changes by guest.
+ * The event might not fire if the client has subscribed to it in onchangeonly
+ * mode, hence the bool_t return type for control register write events.
+ */
+bool_t hvm_event_cr(unsigned int index, unsigned long value,
+                    unsigned long old);
 #define hvm_event_crX(what, new, old) \
     hvm_event_cr(VM_EVENT_X86_##what, new, old)
 void hvm_event_msr(unsigned int msr, uint64_t value);
diff --git a/xen/include/asm-x86/hvm/support.h b/xen/include/asm-x86/hvm/support.h
index 05ef5c5..95d3bb2 100644
--- a/xen/include/asm-x86/hvm/support.h
+++ b/xen/include/asm-x86/hvm/support.h
@@ -124,11 +124,12 @@ void hvm_shadow_handle_cd(struct vcpu *v, unsigned long value);
 
 /* These functions all return X86EMUL return codes. */
 int hvm_set_efer(uint64_t value);
-int hvm_set_cr0(unsigned long value);
-int hvm_set_cr3(unsigned long value);
-int hvm_set_cr4(unsigned long value);
+int hvm_set_cr0(unsigned long value, bool_t may_defer);
+int hvm_set_cr3(unsigned long value, bool_t may_defer);
+int hvm_set_cr4(unsigned long value, bool_t may_defer);
 int hvm_msr_read_intercept(unsigned int msr, uint64_t *msr_content);
-int hvm_msr_write_intercept(unsigned int msr, uint64_t msr_content);
+int hvm_msr_write_intercept(
+    unsigned int msr, uint64_t msr_content, bool_t may_defer);
 int hvm_mov_to_cr(unsigned int cr, unsigned int gpr);
 int hvm_mov_from_cr(unsigned int cr, unsigned int gpr);
 
diff --git a/xen/include/asm-x86/vm_event.h b/xen/include/asm-x86/vm_event.h
index 3881783..2bcfe26 100644
--- a/xen/include/asm-x86/vm_event.h
+++ b/xen/include/asm-x86/vm_event.h
@@ -21,6 +21,7 @@
 #define __ASM_X86_VM_EVENT_H__
 
 #include <xen/sched.h>
+#include <xen/vm_event.h>
 
 int vm_event_init_domain(struct domain *d);
 
@@ -28,4 +29,6 @@ void vm_event_cleanup_domain(struct domain *d);
 
 void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v);
 
+void vm_event_register_write_resume(struct vcpu *v, vm_event_response_t *rsp);
+
 #endif /* __ASM_X86_VM_EVENT_H__ */
diff --git a/xen/include/public/vm_event.h b/xen/include/public/vm_event.h
index f889139..fbc76b2 100644
--- a/xen/include/public/vm_event.h
+++ b/xen/include/public/vm_event.h
@@ -74,6 +74,11 @@
  * VM_EVENT_FLAG_SET_EMUL_READ_DATA are set, only the latter will be honored).
  */
 #define VM_EVENT_FLAG_SET_EMUL_READ_DATA (1 << 5)
+ /*
+  * Deny completion of the operation that triggered the event.
+  * Currently only useful for MSR, CR0, CR3 and CR4 write events.
+  */
+#define VM_EVENT_FLAG_DENY               (1 << 6)
 
 /*
  * Reasons for the vm event request
-- 
1.7.9.5

Re: [PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Jan Beulich]

>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
> Changes since V4:
>  - Rebased the patch to take into account Tamas' "x86/vm_event:
>    toggle singlestep from vm_event response".
>  - Moved the management of heap-allocated vm_event-related
>    domain data to the specific x86 vm_event domain init / cleanup
>    helpers.

I would have understood this to mean vm_event_cleanup_domain()
instead of ...

> --- a/xen/arch/x86/domain.c
> +++ b/xen/arch/x86/domain.c
> @@ -667,6 +667,8 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags,
>  
>  void arch_domain_destroy(struct domain *d)
>  {
> +    xfree(d->arch.event_write_data);

... here. And I see it is being done there, so why also here?

> --- a/xen/arch/x86/vm_event.c
> +++ b/xen/arch/x86/vm_event.c
> @@ -22,11 +22,19 @@
>  
>  #include <xen/sched.h>
>  #include <asm/hvm/hvm.h>
> +#include <asm/vm_event.h>
>  
>  int vm_event_init_domain(struct domain *d)
>  {
>      struct vcpu *v;
>  
> +    if ( !d->arch.event_write_data )
> +        d->arch.event_write_data = xzalloc_array(struct monitor_write_data,
> +                                                 d->max_vcpus);

Looking at this again I wonder why the data isn't being made part of
struct arch_vcpu's vm_event sub-structure. That would also address
the complaint I have here about this not being a guaranteed maximum
page size runtime allocation.

> @@ -61,6 +72,38 @@ void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v)
>      hvm_toggle_singlestep(v);
>  }
>  
> +void vm_event_register_write_resume(struct vcpu *v, vm_event_response_t *rsp)
> +{
> +    if ( rsp->flags & VM_EVENT_FLAG_DENY )
> +    {
> +        struct monitor_write_data *w =
> +            &v->domain->arch.event_write_data[v->vcpu_id];

That would also eliminate this strange construct.

> --- a/xen/include/public/vm_event.h
> +++ b/xen/include/public/vm_event.h
> @@ -74,6 +74,11 @@
>   * VM_EVENT_FLAG_SET_EMUL_READ_DATA are set, only the latter will be honored).
>   */
>  #define VM_EVENT_FLAG_SET_EMUL_READ_DATA (1 << 5)
> + /*
> +  * Deny completion of the operation that triggered the event.
> +  * Currently only useful for MSR, CR0, CR3 and CR4 write events.
> +  */
> +#define VM_EVENT_FLAG_DENY               (1 << 6)

Wouldn't this want adding to the get-capabilities sub-op too?

Jan

Re: [PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Razvan Cojocaru]

On 07/14/2015 03:35 PM, Jan Beulich wrote:
>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>> Changes since V4:
>>  - Rebased the patch to take into account Tamas' "x86/vm_event:
>>    toggle singlestep from vm_event response".
>>  - Moved the management of heap-allocated vm_event-related
>>    domain data to the specific x86 vm_event domain init / cleanup
>>    helpers.
> 
> I would have understood this to mean vm_event_cleanup_domain()
> instead of ...
> 
>> --- a/xen/arch/x86/domain.c
>> +++ b/xen/arch/x86/domain.c
>> @@ -667,6 +667,8 @@ int arch_domain_create(struct domain *d, unsigned int domcr_flags,
>>  
>>  void arch_domain_destroy(struct domain *d)
>>  {
>> +    xfree(d->arch.event_write_data);
> 
> ... here. And I see it is being done there, so why also here?

I was not convinced that it's safe to assume that
vm_event_cleanup_domain() always gets called on domain destruction
(please see also the reply to the 1/3 patch review). That's quite likely
a wrong assumption, but if it's not it's safer.

>> --- a/xen/arch/x86/vm_event.c
>> +++ b/xen/arch/x86/vm_event.c
>> @@ -22,11 +22,19 @@
>>  
>>  #include <xen/sched.h>
>>  #include <asm/hvm/hvm.h>
>> +#include <asm/vm_event.h>
>>  
>>  int vm_event_init_domain(struct domain *d)
>>  {
>>      struct vcpu *v;
>>  
>> +    if ( !d->arch.event_write_data )
>> +        d->arch.event_write_data = xzalloc_array(struct monitor_write_data,
>> +                                                 d->max_vcpus);
> 
> Looking at this again I wonder why the data isn't being made part of
> struct arch_vcpu's vm_event sub-structure. That would also address
> the complaint I have here about this not being a guaranteed maximum
> page size runtime allocation.

I think this is just how the initial suggestion was worded, I'll change it.

>> @@ -61,6 +72,38 @@ void vm_event_toggle_singlestep(struct domain *d, struct vcpu *v)
>>      hvm_toggle_singlestep(v);
>>  }
>>  
>> +void vm_event_register_write_resume(struct vcpu *v, vm_event_response_t *rsp)
>> +{
>> +    if ( rsp->flags & VM_EVENT_FLAG_DENY )
>> +    {
>> +        struct monitor_write_data *w =
>> +            &v->domain->arch.event_write_data[v->vcpu_id];
> 
> That would also eliminate this strange construct.

Indeed, I'm not a fan of it either.

>> --- a/xen/include/public/vm_event.h
>> +++ b/xen/include/public/vm_event.h
>> @@ -74,6 +74,11 @@
>>   * VM_EVENT_FLAG_SET_EMUL_READ_DATA are set, only the latter will be honored).
>>   */
>>  #define VM_EVENT_FLAG_SET_EMUL_READ_DATA (1 << 5)
>> + /*
>> +  * Deny completion of the operation that triggered the event.
>> +  * Currently only useful for MSR, CR0, CR3 and CR4 write events.
>> +  */
>> +#define VM_EVENT_FLAG_DENY               (1 << 6)
> 
> Wouldn't this want adding to the get-capabilities sub-op too?

Yes, it's best to add it. Ack.


Thanks,
Razvan

Re: [PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Jan Beulich]

>>> On 14.07.15 at 15:45, <rcojocaru@bitdefender.com> wrote:
> On 07/14/2015 03:35 PM, Jan Beulich wrote:
>>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>>> --- a/xen/arch/x86/vm_event.c
>>> +++ b/xen/arch/x86/vm_event.c
>>> @@ -22,11 +22,19 @@
>>>  
>>>  #include <xen/sched.h>
>>>  #include <asm/hvm/hvm.h>
>>> +#include <asm/vm_event.h>
>>>  
>>>  int vm_event_init_domain(struct domain *d)
>>>  {
>>>      struct vcpu *v;
>>>  
>>> +    if ( !d->arch.event_write_data )
>>> +        d->arch.event_write_data = xzalloc_array(struct monitor_write_data,
>>> +                                                 d->max_vcpus);
>> 
>> Looking at this again I wonder why the data isn't being made part of
>> struct arch_vcpu's vm_event sub-structure. That would also address
>> the complaint I have here about this not being a guaranteed maximum
>> page size runtime allocation.
> 
> I think this is just how the initial suggestion was worded, I'll change it.

Right - after having sent the reply I started wondering whether
maybe I had asked for this. But if I did, then surely not with
xzalloc_array(), but vzalloc().

If you moved this into struct arch_vcpu (again), then its size would
likely call for the whole vm_event structure to become indirectly
accessed and allocated.

Jan

Re: [PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Razvan Cojocaru]

On 07/14/2015 05:41 PM, Jan Beulich wrote:
>>>> On 14.07.15 at 15:45, <rcojocaru@bitdefender.com> wrote:
>> On 07/14/2015 03:35 PM, Jan Beulich wrote:
>>>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>>>> --- a/xen/arch/x86/vm_event.c
>>>> +++ b/xen/arch/x86/vm_event.c
>>>> @@ -22,11 +22,19 @@
>>>>  
>>>>  #include <xen/sched.h>
>>>>  #include <asm/hvm/hvm.h>
>>>> +#include <asm/vm_event.h>
>>>>  
>>>>  int vm_event_init_domain(struct domain *d)
>>>>  {
>>>>      struct vcpu *v;
>>>>  
>>>> +    if ( !d->arch.event_write_data )
>>>> +        d->arch.event_write_data = xzalloc_array(struct monitor_write_data,
>>>> +                                                 d->max_vcpus);
>>>
>>> Looking at this again I wonder why the data isn't being made part of
>>> struct arch_vcpu's vm_event sub-structure. That would also address
>>> the complaint I have here about this not being a guaranteed maximum
>>> page size runtime allocation.
>>
>> I think this is just how the initial suggestion was worded, I'll change it.
> 
> Right - after having sent the reply I started wondering whether
> maybe I had asked for this. But if I did, then surely not with
> xzalloc_array(), but vzalloc().
> 
> If you moved this into struct arch_vcpu (again), then its size would
> likely call for the whole vm_event structure to become indirectly
> accessed and allocated.

In that case would it suffice to just switch to vzalloc() in this case?

I'm not opposed to just placing all the data (this and the
memory-content hiding data) in struct vm_event and allocate that as a
whole, but that would change patch 1/3, 3/3 and also touch other code.


Thanks,
Razvan

Re: [PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Jan Beulich]

>>> On 14.07.15 at 17:04, <rcojocaru@bitdefender.com> wrote:
> On 07/14/2015 05:41 PM, Jan Beulich wrote:
>>>>> On 14.07.15 at 15:45, <rcojocaru@bitdefender.com> wrote:
>>> On 07/14/2015 03:35 PM, Jan Beulich wrote:
>>>>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>>>>> --- a/xen/arch/x86/vm_event.c
>>>>> +++ b/xen/arch/x86/vm_event.c
>>>>> @@ -22,11 +22,19 @@
>>>>>  
>>>>>  #include <xen/sched.h>
>>>>>  #include <asm/hvm/hvm.h>
>>>>> +#include <asm/vm_event.h>
>>>>>  
>>>>>  int vm_event_init_domain(struct domain *d)
>>>>>  {
>>>>>      struct vcpu *v;
>>>>>  
>>>>> +    if ( !d->arch.event_write_data )
>>>>> +        d->arch.event_write_data = xzalloc_array(struct monitor_write_data,
>>>>> +                                                 d->max_vcpus);
>>>>
>>>> Looking at this again I wonder why the data isn't being made part of
>>>> struct arch_vcpu's vm_event sub-structure. That would also address
>>>> the complaint I have here about this not being a guaranteed maximum
>>>> page size runtime allocation.
>>>
>>> I think this is just how the initial suggestion was worded, I'll change it.
>> 
>> Right - after having sent the reply I started wondering whether
>> maybe I had asked for this. But if I did, then surely not with
>> xzalloc_array(), but vzalloc().
>> 
>> If you moved this into struct arch_vcpu (again), then its size would
>> likely call for the whole vm_event structure to become indirectly
>> accessed and allocated.
> 
> In that case would it suffice to just switch to vzalloc() in this case?

For now, yes.

> I'm not opposed to just placing all the data (this and the
> memory-content hiding data) in struct vm_event and allocate that as a
> whole, but that would change patch 1/3, 3/3 and also touch other code.

Would be a nice thing to clean up post-4.6.

Jan

Re: [PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Razvan Cojocaru]

On 07/14/2015 06:55 PM, Jan Beulich wrote:
>>>> On 14.07.15 at 17:04, <rcojocaru@bitdefender.com> wrote:
>> On 07/14/2015 05:41 PM, Jan Beulich wrote:
>>>>>> On 14.07.15 at 15:45, <rcojocaru@bitdefender.com> wrote:
>>>> On 07/14/2015 03:35 PM, Jan Beulich wrote:
>>>>>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>>>>>> --- a/xen/arch/x86/vm_event.c
>>>>>> +++ b/xen/arch/x86/vm_event.c
>>>>>> @@ -22,11 +22,19 @@
>>>>>>  
>>>>>>  #include <xen/sched.h>
>>>>>>  #include <asm/hvm/hvm.h>
>>>>>> +#include <asm/vm_event.h>
>>>>>>  
>>>>>>  int vm_event_init_domain(struct domain *d)
>>>>>>  {
>>>>>>      struct vcpu *v;
>>>>>>  
>>>>>> +    if ( !d->arch.event_write_data )
>>>>>> +        d->arch.event_write_data = xzalloc_array(struct monitor_write_data,
>>>>>> +                                                 d->max_vcpus);
>>>>>
>>>>> Looking at this again I wonder why the data isn't being made part of
>>>>> struct arch_vcpu's vm_event sub-structure. That would also address
>>>>> the complaint I have here about this not being a guaranteed maximum
>>>>> page size runtime allocation.
>>>>
>>>> I think this is just how the initial suggestion was worded, I'll change it.
>>>
>>> Right - after having sent the reply I started wondering whether
>>> maybe I had asked for this. But if I did, then surely not with
>>> xzalloc_array(), but vzalloc().
>>>
>>> If you moved this into struct arch_vcpu (again), then its size would
>>> likely call for the whole vm_event structure to become indirectly
>>> accessed and allocated.
>>
>> In that case would it suffice to just switch to vzalloc() in this case?
> 
> For now, yes.
> 
>> I'm not opposed to just placing all the data (this and the
>> memory-content hiding data) in struct vm_event and allocate that as a
>> whole, but that would change patch 1/3, 3/3 and also touch other code.
> 
> Would be a nice thing to clean up post-4.6.

I agree, and I will add it to my queue.


Thanks,
Razvan

Re: [PATCH V5 3/3] xen/vm_event: Deny register writes if refused by vm_event reply

[Razvan Cojocaru]

On 07/14/2015 03:35 PM, Jan Beulich wrote:
>> --- a/xen/include/public/vm_event.h
>> +++ b/xen/include/public/vm_event.h
>> @@ -74,6 +74,11 @@
>>   * VM_EVENT_FLAG_SET_EMUL_READ_DATA are set, only the latter will be honored).
>>   */
>>  #define VM_EVENT_FLAG_SET_EMUL_READ_DATA (1 << 5)
>> + /*
>> +  * Deny completion of the operation that triggered the event.
>> +  * Currently only useful for MSR, CR0, CR3 and CR4 write events.
>> +  */
>> +#define VM_EVENT_FLAG_DENY               (1 << 6)
> 
> Wouldn't this want adding to the get-capabilities sub-op too?

Actually, no (sorry for rushing with the answer before), the monitor.c
capabilities issue refers to main event types, whereas this is a
response flag.

 45 static inline uint32_t get_capabilities(struct domain *d)
 46 {
 47     uint32_t capabilities = 0;
 48
 49     /*
 50      * At the moment only Intel HVM domains are supported. However,
event
 51      * delivery could be extended to AMD and PV domains.
 52      */
 53     if ( !is_hvm_domain(d) || !cpu_has_vmx )
 54         return capabilities;
 55
 56     capabilities = (1 << XEN_DOMCTL_MONITOR_EVENT_WRITE_CTRLREG) |
 57                    (1 << XEN_DOMCTL_MONITOR_EVENT_MOV_TO_MSR) |
 58                    (1 << XEN_DOMCTL_MONITOR_EVENT_SOFTWARE_BREAKPOINT) |
 59                    (1 << XEN_DOMCTL_MONITOR_EVENT_GUEST_REQUEST);
 60
 61     /* Since we know this is on VMX, we can just call the hvm func */
 62     if ( hvm_is_singlestep_supported() )
 63         capabilities |= (1 << XEN_DOMCTL_MONITOR_EVENT_SINGLESTEP);
 64
 65     return capabilities;
 66 }

So if we have the (1 << XEN_DOMCTL_MONITOR_EVENT_WRITE_CTRLREG) and (1
<< XEN_DOMCTL_MONITOR_EVENT_MOV_TO_MSR), that also means we can deny
those writes (by setting the VM_EVENT_FLAG_DENY in the vm_event
response), no additional checks are necessary.


Thanks,
Razvan

Re: [PATCH V5 0/3] Vm_event memory introspection helpers

[Jan Beulich]

>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
> I've also moved x86 logic in patch 3/3 to x86 source files, this
> seems to have gone unnoticed but would likely have not compiled
> on ARM.

Which leaves open whether this time you actually checked that
ARM continues to build.

Jan

Re: [PATCH V5 0/3] Vm_event memory introspection helpers

[Razvan Cojocaru]

On 07/14/2015 01:50 PM, Jan Beulich wrote:
>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>> I've also moved x86 logic in patch 3/3 to x86 source files, this
>> seems to have gone unnoticed but would likely have not compiled
>> on ARM.
> 
> Which leaves open whether this time you actually checked that
> ARM continues to build.

I did check, and again just now. My patches don't break the ARM build,
but I just found that on my ARM system, current Xen staging doesn't
build: http://pastebin.com/RnywiCX7

# cat /etc/issue
Ubuntu 14.04 LTS \n \l

# uname -a
Linux localhost.localdomain 3.15.0-rc8+ #3 SMP PREEMPT Fri Jan 30
14:08:23 EET 2015 aarch64 aarch64 aarch64 GNU/Linux

# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/4.9/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro
4.9.2-10ubuntu13'
--with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs
--enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.9 --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib
--enable-nls --with-sysroot=/ --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes
--enable-gnu-unique-object --disable-libsanitizer --disable-libquadmath
--enable-plugin --with-system-zlib --disable-browser-plugin
--enable-java-awt=gtk --enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-arm64/jre
--enable-java-home
--with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-arm64
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-arm64
--with-arch-directory=arm64
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-multiarch
--disable-werror --enable-checking=release --build=aarch64-linux-gnu
--host=aarch64-linux-gnu --target=aarch64-linux-gnu
Thread model: posix
gcc version 4.9.2 (Ubuntu/Linaro 4.9.2-10ubuntu13)

I've temporarily fixed that by wrapping the offending #defines in
"#ifndef constant" statements, so that I could continue to check that
the build works, but that's likely not the proper fix.


Hope this helps,
Razvan

Re: [PATCH V5 0/3] Vm_event memory introspection helpers

[Jan Beulich]

>>> On 14.07.15 at 13:45, <rcojocaru@bitdefender.com> wrote:
> On 07/14/2015 01:50 PM, Jan Beulich wrote:
>>>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
>>> I've also moved x86 logic in patch 3/3 to x86 source files, this
>>> seems to have gone unnoticed but would likely have not compiled
>>> on ARM.
>> 
>> Which leaves open whether this time you actually checked that
>> ARM continues to build.
> 
> I did check, and again just now. My patches don't break the ARM build,
> but I just found that on my ARM system, current Xen staging doesn't
> build: http://pastebin.com/RnywiCX7 

That's the tool stack having issues, while we're talking about the
hypervisor build here.

Jan

Re: [PATCH V5 0/3] Vm_event memory introspection helpers

[Ian Campbell]

On Tue, 2015-07-14 at 14:45 +0300, Razvan Cojocaru wrote:
> On 07/14/2015 01:50 PM, Jan Beulich wrote:
> >>>> On 13.07.15 at 19:14, <rcojocaru@bitdefender.com> wrote:
> >> I've also moved x86 logic in patch 3/3 to x86 source files, this
> >> seems to have gone unnoticed but would likely have not compiled
> >> on ARM.
> > 
> > Which leaves open whether this time you actually checked that
> > ARM continues to build.
> 
> I did check, and again just now. My patches don't break the ARM build,
> but I just found that on my ARM system, current Xen staging doesn't
> build: http://pastebin.com/RnywiCX7

This is a glibc error with your distro, like this
https://bugs.launchpad.net/linaro-aarch64/+bug/1169164

> I've temporarily fixed that by wrapping the offending #defines in
> "#ifndef constant" statements, so that I could continue to check that
> the build works, but that's likely not the proper fix.

Indeed, it the hack which works if you are unable to upgrade your libc
for some reason. FWIW I just applied the patches from that bug to the
installed copies of the relevant headers, which is certainly skanky...

Ian.