[RFC PATCH 0/18] Xenstore stub domain

[RFC PATCH 0/18] Xenstore stub domain

[Daniel De Graaf]

This patch series allows xenstored to run in a stub domian started by
dom0. It is based on a patch series posted by Alex Zeffertt in 2009 -
http://old-list-archives.xen.org/archives/html/xen-devel/2009-03/msg01488.html


A domain configuration for starting xenstored looks like:

kernel='/home/daniel/xen/stubdom/mini-os-x86_64-xenstore/mini-os'
extra=''
memory=50
name='xenstore'

Once xenstore is started, "xenstore_dom=1" needs to be added to other
domain's configurations in order to set up the xenstore connection to
domain 1.

The following program handles post-creation parts of xenstored. To use
it, run "xl create -p xenstore" and then "init-xenstore $domid". The
running xenstored must be stopped to prevent xl using the UNIX sockets,
and xenconsoled needs to be restarted after switching xenstores.

/* init-xenstore.c: link with -lxenctrl */

#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <stdlib.h>
#include <sys/ioctl.h>
#include <sys/mman.h>

#define __XEN_TOOLS__
#include <xen/domctl.h>
#include "xenctrl.h"

#define IOCTL_XENBUS_BACKEND_SETUP _IOC(_IOC_NONE, 'B', 1, 0)
#define IOCTL_XENBUS_BACKEND_COMMIT _IOC(_IOC_NONE, 'B', 2, 0)

static void set_virq(int domid, int virq)
{
	struct xen_domctl command;
	xc_interface *xch;

	xch = xc_interface_open(NULL, NULL, 0);

	memset(&command, 0, sizeof(command)); 
	command.cmd               = XEN_DOMCTL_set_virq_handler;
	command.interface_version = XEN_DOMCTL_INTERFACE_VERSION;
	command.domain            = domid;
	command.u.set_virq_handler.virq = virq;
	xc_domctl(xch, &command);
	xc_interface_close(xch);
}

int main(int argc, char** argv)
{
	char buf[512];
	int domid = atoi(argv[1]);

	set_virq(domid, VIRQ_DOM_EXC);

	int fd = open("/dev/xen/xenbus_backend", O_RDWR);
	void *map = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
	int rv = ioctl(fd, IOCTL_XENBUS_BACKEND_SETUP, domid);
	*(uint16_t*)(map + 0x810) = rv;
	snprintf(buf, 512, "xl unpause %d", domid);
	system(buf);
	ioctl(fd, IOCTL_XENBUS_BACKEND_COMMIT, 0);
	return 0;
}

-------------------------------------------------

Dom0 kernel changes:
    [PATCH] xenbus: Add support for xenbus backend in stub domain

This is based on the new /dev/xen devices introduced in Linux 3.3.

Hypervisor changes:
    [PATCH 01/18] xen: reinstate previously unused
    [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to
    [PATCH 03/18] xsm: allow use of XEN_DOMCTL_getdomaininfo by
    [PATCH 04/18] xen: Preserve reserved grant entries when switching

Patch 1 & 4 are required for setting up grant entries in new domains.
Patch 2 & 3 allow xenstored to run in an unprivileged domain. This
currently requires XSM to be enabled to avoid allowing all domUs access
to XEN_DOMCTL_getdomaininfo, so the patch only allows this hypercall if
XSM is being compiled in.

Toolstack changes:
    [PATCH 05/18] tools/libxl: Add xenstore and console backend domain
    [PATCH 06/18] lib{xc,xl}: Seed grant tables with xenstore and

These patches populate two of the eight reserved grant entries in new
domains with the xenstore and console shared pages, which is required
if xenstored is not run in a privileged domain.

Minios and xenstored:
    [PATCH 07/18] mini-os: avoid crash if no console is provided
    [PATCH 08/18] mini-os: avoid crash if no xenstore is provided
    [PATCH 09/18] mini-os: remove per-fd evtchn limit
    [PATCH 10/18] xenstored: use grant references instead of
    [PATCH 11/18] xenstored: add NO_SOCKETS compilation option
    [PATCH 12/18] xenstored support for in-memory rather than FS based
    [PATCH 13/18] xenstored: support running in minios stubdom
    [PATCH 14/18] xenstored: always use xc_gnttab_munmap in stubdom
    [PATCH 15/18] xenstored: add --event parameter for bootstrapping
    [PATCH 16/18] xenstored: pull dom0 event port from shared page
    [PATCH 17/18] xenstored: use domain_is_unprivileged instead of
    [PATCH 18/18] xenstored: add --priv-domid parameter

Support for running in a stub domain

[PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

This patch reinstates the XENMEM_remove_from_physmap hypercall
which was removed in 19041:ee62aaafff46 because it was not used.

However, is now needed in order to support xenstored stub domains.
The xenstored stub domain is not priviliged like dom0 and so cannot
unilaterally map the xenbus page of other guests into it's address
space.  Therefore, before creating a domU the domain builder needs to
seed its grant table with a grant ref allowing the xenstored stub
domain to access the new domU's xenbus page.

At present domU's do not start with their grant table mapped.
Instead it gets mapped when the guest requests a grant table from
the hypervisor.

In order to seed the grant table, the domain builder first needs to
map it into dom0 address space.  But the hypercall to do this
requires a gpfn (guest pfn), which is an mfn for PV guest, but a pfn
for HVM guests.  Therfore, in order to seed the grant table of an
HVM guest, dom0 needs to *temporarily* map it into the guest's
"physical" address space.

Hence the need to reinstate the XENMEM_remove_from_physmap hypercall.

Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 xen/arch/ia64/xen/mm.c          |   34 ++++++++++++++++++++++++++++++++++
 xen/arch/x86/mm.c               |   35 +++++++++++++++++++++++++++++++++++
 xen/arch/x86/x86_64/compat/mm.c |   14 ++++++++++++++
 xen/include/public/memory.h     |   16 ++++++++++++++++
 xen/include/xlat.lst            |    1 +
 xen/include/xsm/xsm.h           |    6 ++++++
 xen/xsm/dummy.c                 |    6 ++++++
 xen/xsm/flask/hooks.c           |    6 ++++++
 8 files changed, 118 insertions(+), 0 deletions(-)

diff --git a/xen/arch/ia64/xen/mm.c b/xen/arch/ia64/xen/mm.c
index 694bf95..1b42e25 100644
--- a/xen/arch/ia64/xen/mm.c
+++ b/xen/arch/ia64/xen/mm.c
@@ -3447,6 +3447,40 @@ arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
         break;
     }
 
+    case XENMEM_remove_from_physmap:
+    {
+        struct xen_remove_from_physmap xrfp;
+        unsigned long mfn;
+        struct domain *d;
+
+        if ( copy_from_guest(&xrfp, arg, 1) )
+            return -EFAULT;
+
+        rc = rcu_lock_target_domain_by_id(xrfp.domid, &d);
+        if ( rc != 0 )
+            return rc;
+
+        if ( xsm_remove_from_physmap(current->domain, d) )
+        {
+            rcu_unlock_domain(d);
+            return -EPERM;
+        }
+
+        domain_lock(d);
+
+        mfn = gmfn_to_mfn(d, xrfp.gpfn);
+
+        if ( mfn_valid(mfn) )
+            guest_physmap_remove_page(d, xrfp.gpfn, mfn, 0);
+
+        domain_unlock(d);
+
+        rcu_unlock_domain(d);
+
+        break;
+    }
+
+
     case XENMEM_machine_memory_map:
     {
         struct xen_memory_map memmap;
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 52222dd..0b5332b 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -4871,6 +4871,41 @@ long arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
         return rc;
     }
 
+    case XENMEM_remove_from_physmap:
+    {
+        struct xen_remove_from_physmap xrfp;
+        unsigned long mfn;
+        struct domain *d;
+
+        if ( copy_from_guest(&xrfp, arg, 1) )
+            return -EFAULT;
+
+        rc = rcu_lock_target_domain_by_id(xrfp.domid, &d);
+        if ( rc != 0 )
+            return rc;
+
+        if ( xsm_remove_from_physmap(current->domain, d) )
+        {
+            rcu_unlock_domain(d);
+            return -EPERM;
+        }
+
+        domain_lock(d);
+
+        mfn = get_gfn_untyped(d, xrfp.gpfn);
+
+        if ( mfn_valid(mfn) )
+            guest_physmap_remove_page(d, xrfp.gpfn, mfn, PAGE_ORDER_4K);
+
+        put_gfn(d, xrfp.gpfn);
+
+        domain_unlock(d);
+
+        rcu_unlock_domain(d);
+
+        break;
+    }
+
     case XENMEM_set_memory_map:
     {
         struct xen_foreign_memory_map fmap;
diff --git a/xen/arch/x86/x86_64/compat/mm.c b/xen/arch/x86/x86_64/compat/mm.c
index bea94fe..dde5430 100644
--- a/xen/arch/x86/x86_64/compat/mm.c
+++ b/xen/arch/x86/x86_64/compat/mm.c
@@ -82,6 +82,20 @@ int compat_arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
         break;
     }
 
+    case XENMEM_remove_from_physmap:
+    {
+        struct compat_remove_from_physmap cmp;
+        struct xen_remove_from_physmap *nat = (void *)COMPAT_ARG_XLAT_VIRT_BASE;
+
+        if ( copy_from_guest(&cmp, arg, 1) )
+            return -EFAULT;
+
+        XLAT_remove_from_physmap(nat, &cmp);
+        rc = arch_memory_op(op, guest_handle_from_ptr(nat, void));
+
+        break;
+    }
+
     case XENMEM_set_memory_map:
     {
         struct compat_foreign_memory_map cmp;
diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
index c5b78a8..308deff 100644
--- a/xen/include/public/memory.h
+++ b/xen/include/public/memory.h
@@ -229,6 +229,22 @@ struct xen_add_to_physmap {
 typedef struct xen_add_to_physmap xen_add_to_physmap_t;
 DEFINE_XEN_GUEST_HANDLE(xen_add_to_physmap_t);
 
+/*
+ * Unmaps the page appearing at a particular GPFN from the specified guest's
+ * pseudophysical address space.
+ * arg == addr of xen_remove_from_physmap_t.
+ */
+#define XENMEM_remove_from_physmap      15
+struct xen_remove_from_physmap {
+    /* Which domain to change the mapping for. */
+    domid_t domid;
+
+    /* GPFN of the current mapping of the page. */
+    xen_pfn_t     gpfn;
+};
+typedef struct xen_remove_from_physmap xen_remove_from_physmap_t;
+DEFINE_XEN_GUEST_HANDLE(xen_remove_from_physmap_t);
+
 /*** REMOVED ***/
 /*#define XENMEM_translate_gpfn_list  8*/
 
diff --git a/xen/include/xlat.lst b/xen/include/xlat.lst
index 3d92175..9e67259 100644
--- a/xen/include/xlat.lst
+++ b/xen/include/xlat.lst
@@ -54,6 +54,7 @@
 !	kexec_image			kexec.h
 !	kexec_range			kexec.h
 !	add_to_physmap			memory.h
+!	remove_from_physmap		memory.h
 !	foreign_memory_map		memory.h
 !	memory_exchange			memory.h
 !	memory_map			memory.h
diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
index df6cec2..566c808 100644
--- a/xen/include/xsm/xsm.h
+++ b/xen/include/xsm/xsm.h
@@ -169,6 +169,7 @@ struct xsm_operations {
     int (*update_va_mapping) (struct domain *d, struct domain *f, 
                                                             l1_pgentry_t pte);
     int (*add_to_physmap) (struct domain *d1, struct domain *d2);
+    int (*remove_from_physmap) (struct domain *d1, struct domain *d2);
     int (*sendtrigger) (struct domain *d);
     int (*bind_pt_irq) (struct domain *d, struct xen_domctl_bind_pt_irq *bind);
     int (*unbind_pt_irq) (struct domain *d);
@@ -738,6 +739,11 @@ static inline int xsm_add_to_physmap(struct domain *d1, struct domain *d2)
     return xsm_call(add_to_physmap(d1, d2));
 }
 
+static inline int xsm_remove_from_physmap(struct domain *d1, struct domain *d2)
+{
+    return xsm_call(remove_from_physmap(d1, d2));
+}
+
 static inline int xsm_sendtrigger(struct domain *d)
 {
     return xsm_call(sendtrigger(d));
diff --git a/xen/xsm/dummy.c b/xen/xsm/dummy.c
index 4bbfbff..65daa4e 100644
--- a/xen/xsm/dummy.c
+++ b/xen/xsm/dummy.c
@@ -529,6 +529,11 @@ static int dummy_add_to_physmap (struct domain *d1, struct domain *d2)
     return 0;
 }
 
+static int dummy_remove_from_physmap (struct domain *d1, struct domain *d2)
+{
+    return 0;
+}
+
 static int dummy_sendtrigger (struct domain *d)
 {
     return 0;
@@ -690,6 +695,7 @@ void xsm_fixup_ops (struct xsm_operations *ops)
     set_to_dummy_if_null(ops, mmu_machphys_update);
     set_to_dummy_if_null(ops, update_va_mapping);
     set_to_dummy_if_null(ops, add_to_physmap);
+    set_to_dummy_if_null(ops, remove_from_physmap);
     set_to_dummy_if_null(ops, sendtrigger);
     set_to_dummy_if_null(ops, bind_pt_irq);
     set_to_dummy_if_null(ops, pin_mem_cacheattr);
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index 0d35767..a2020a9 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -1283,6 +1283,11 @@ static int flask_add_to_physmap(struct domain *d1, struct domain *d2)
     return domain_has_perm(d1, d2, SECCLASS_MMU, MMU__PHYSMAP);
 }
 
+static int flask_remove_from_physmap(struct domain *d1, struct domain *d2)
+{
+    return domain_has_perm(d1, d2, SECCLASS_MMU, MMU__PHYSMAP);
+}
+
 static int flask_sendtrigger(struct domain *d)
 {
     return domain_has_perm(current->domain, d, SECCLASS_DOMAIN, DOMAIN__TRIGGER);
@@ -1550,6 +1555,7 @@ static struct xsm_operations flask_ops = {
     .mmu_machphys_update = flask_mmu_machphys_update,
     .update_va_mapping = flask_update_va_mapping,
     .add_to_physmap = flask_add_to_physmap,
+    .remove_from_physmap = flask_remove_from_physmap,
     .sendtrigger = flask_sendtrigger,
     .get_device_group = flask_get_device_group,
     .test_assign_device = flask_test_assign_device,
-- 
1.7.7.5

Re: [PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Jan Beulich]

>>> On 11.01.12 at 18:21, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> --- a/xen/include/xlat.lst
> +++ b/xen/include/xlat.lst
> @@ -54,6 +54,7 @@
>  !	kexec_image			kexec.h
>  !	kexec_range			kexec.h
>  !	add_to_physmap			memory.h
> +!	remove_from_physmap		memory.h
>  !	foreign_memory_map		memory.h
>  !	memory_exchange			memory.h
>  !	memory_map			memory.h

Alphabetically, please.

Jan

[PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Daniel De Graaf]

This patch sends global VIRQs to a domain designated as the VIRQ handler
instead of sending all global VIRQ events to dom0. This is required in
order to run xenstored in a stubdom, because VIRQ_DOM_EXC must be sent
to xenstored for domain destruction to work properly.

This patch was inspired by the xenstored stubdomain patch series sent to
xen-devel by Alex Zeffertt in 2009.

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/flask/policy/policy/flask/access_vectors |    1 +
 xen/arch/x86/cpu/mcheck/amd_nonfatal.c         |    2 +-
 xen/arch/x86/cpu/mcheck/mce.c                  |    2 +-
 xen/arch/x86/cpu/mcheck/mce_intel.c            |    6 +-
 xen/arch/x86/cpu/mcheck/non-fatal.c            |    2 +-
 xen/common/cpu.c                               |    4 +-
 xen/common/domain.c                            |   10 ++--
 xen/common/domctl.c                            |   17 ++++++
 xen/common/event_channel.c                     |   63 +++++++++++++++++++++++-
 xen/common/trace.c                             |    2 +-
 xen/drivers/char/console.c                     |    4 +-
 xen/include/public/domctl.h                    |    8 +++
 xen/include/xen/event.h                        |   18 ++++++-
 xen/include/xsm/xsm.h                          |    6 ++
 xen/xsm/dummy.c                                |    6 ++
 xen/xsm/flask/hooks.c                          |    6 ++
 xen/xsm/flask/include/av_perm_to_string.h      |    1 +
 xen/xsm/flask/include/av_permissions.h         |    1 +
 18 files changed, 140 insertions(+), 19 deletions(-)

diff --git a/tools/flask/policy/policy/flask/access_vectors b/tools/flask/policy/policy/flask/access_vectors
index 644f2e1..5901911 100644
--- a/tools/flask/policy/policy/flask/access_vectors
+++ b/tools/flask/policy/policy/flask/access_vectors
@@ -85,6 +85,7 @@ class domain
 	getpodtarget
 	setpodtarget
 	set_misc_info
+	set_virq_handler
 }
 
 class hvm
diff --git a/xen/arch/x86/cpu/mcheck/amd_nonfatal.c b/xen/arch/x86/cpu/mcheck/amd_nonfatal.c
index 50288bd..9222098 100644
--- a/xen/arch/x86/cpu/mcheck/amd_nonfatal.c
+++ b/xen/arch/x86/cpu/mcheck/amd_nonfatal.c
@@ -100,7 +100,7 @@ static void mce_amd_checkregs(void *info)
 
 		if (dom0_vmce_enabled()) {
 			mctelem_commit(mctc);
-			send_guest_global_virq(dom0, VIRQ_MCA);
+			send_global_virq(VIRQ_MCA);
 		} else if (++dumpcount >= 10) {
 			x86_mcinfo_dump((struct mc_info *)mctelem_dataptr(mctc));
 			mctelem_dismiss(mctc);
diff --git a/xen/arch/x86/cpu/mcheck/mce.c b/xen/arch/x86/cpu/mcheck/mce.c
index b592041..c4e4477 100644
--- a/xen/arch/x86/cpu/mcheck/mce.c
+++ b/xen/arch/x86/cpu/mcheck/mce.c
@@ -594,7 +594,7 @@ void mcheck_cmn_handler(struct cpu_user_regs *regs, long error_code,
         if (dom0_vmce_enabled()) {
             if (mctc != NULL)
                 mctelem_commit(mctc);
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         } else {
             x86_mcinfo_dump(mci);
             if (mctc != NULL)
diff --git a/xen/arch/x86/cpu/mcheck/mce_intel.c b/xen/arch/x86/cpu/mcheck/mce_intel.c
index 0986025..0894080 100644
--- a/xen/arch/x86/cpu/mcheck/mce_intel.c
+++ b/xen/arch/x86/cpu/mcheck/mce_intel.c
@@ -354,7 +354,7 @@ static void mce_softirq(void)
         /* Step2: Send Log to DOM0 through vIRQ */
         if (dom0_vmce_enabled()) {
             mce_printk(MCE_VERBOSE, "MCE: send MCE# to DOM0 through virq\n");
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         }
     }
 
@@ -1085,7 +1085,7 @@ static void cmci_discover(void)
     if (bs.errcnt && mctc != NULL) {
         if (dom0_vmce_enabled()) {
             mctelem_commit(mctc);
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         } else {
             x86_mcinfo_dump(mctelem_dataptr(mctc));
             mctelem_dismiss(mctc);
@@ -1205,7 +1205,7 @@ fastcall void smp_cmci_interrupt(struct cpu_user_regs *regs)
         if (dom0_vmce_enabled()) {
             mctelem_commit(mctc);
             mce_printk(MCE_VERBOSE, "CMCI: send CMCI to DOM0 through virq\n");
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         } else {
             x86_mcinfo_dump(mctelem_dataptr(mctc));
             mctelem_dismiss(mctc);
diff --git a/xen/arch/x86/cpu/mcheck/non-fatal.c b/xen/arch/x86/cpu/mcheck/non-fatal.c
index c57688f..1dded9b 100644
--- a/xen/arch/x86/cpu/mcheck/non-fatal.c
+++ b/xen/arch/x86/cpu/mcheck/non-fatal.c
@@ -55,7 +55,7 @@ static void mce_checkregs (void *info)
 
 		if (dom0_vmce_enabled()) {
 			mctelem_commit(mctc);
-			send_guest_global_virq(dom0, VIRQ_MCA);
+			send_global_virq(VIRQ_MCA);
 		} else if (++dumpcount >= 10) {
 			x86_mcinfo_dump((struct mc_info *)mctelem_dataptr(mctc));
 			mctelem_dismiss(mctc);
diff --git a/xen/common/cpu.c b/xen/common/cpu.c
index 79abdb7..630881e 100644
--- a/xen/common/cpu.c
+++ b/xen/common/cpu.c
@@ -108,7 +108,7 @@ int cpu_down(unsigned int cpu)
     notifier_rc = notifier_call_chain(&cpu_chain, CPU_DEAD, hcpu, NULL);
     BUG_ON(notifier_rc != NOTIFY_DONE);
 
-    send_guest_global_virq(dom0, VIRQ_PCPU_STATE);
+    send_global_virq(VIRQ_PCPU_STATE);
     cpu_hotplug_done();
     return 0;
 
@@ -148,7 +148,7 @@ int cpu_up(unsigned int cpu)
     notifier_rc = notifier_call_chain(&cpu_chain, CPU_ONLINE, hcpu, NULL);
     BUG_ON(notifier_rc != NOTIFY_DONE);
 
-    send_guest_global_virq(dom0, VIRQ_PCPU_STATE);
+    send_global_virq(VIRQ_PCPU_STATE);
 
     cpu_hotplug_done();
     return 0;
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 52a63ef..c4d98d9 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -116,7 +116,7 @@ static void __domain_finalise_shutdown(struct domain *d)
     if ( (d->shutdown_code == SHUTDOWN_suspend) && d->suspend_evtchn )
         evtchn_send(d, d->suspend_evtchn);
     else
-        send_guest_global_virq(dom0, VIRQ_DOM_EXC);
+        send_global_virq(VIRQ_DOM_EXC);
 }
 
 static void vcpu_check_shutdown(struct vcpu *v)
@@ -492,7 +492,7 @@ int domain_kill(struct domain *d)
         }
         d->is_dying = DOMDYING_dead;
         put_domain(d);
-        send_guest_global_virq(dom0, VIRQ_DOM_EXC);
+        send_global_virq(VIRQ_DOM_EXC);
         /* fallthrough */
     case DOMDYING_dead:
         break;
@@ -633,7 +633,7 @@ void domain_pause_for_debugger(void)
     for_each_vcpu ( d, v )
         vcpu_sleep_nosync(v);
 
-    send_guest_global_virq(dom0, VIRQ_DEBUGGER);
+    send_global_virq(VIRQ_DEBUGGER);
 }
 
 /* Complete domain destroy after RCU readers are not holding old references. */
@@ -659,6 +659,8 @@ static void complete_domain_destroy(struct rcu_head *head)
 
     watchdog_domain_destroy(d);
 
+    clear_global_virq_handlers(d);
+
     rangeset_domain_destroy(d);
 
     cpupool_rm_domain(d);
@@ -690,7 +692,7 @@ static void complete_domain_destroy(struct rcu_head *head)
     free_cpumask_var(d->domain_dirty_cpumask);
     free_domain_struct(d);
 
-    send_guest_global_virq(dom0, VIRQ_DOM_EXC);
+    send_global_virq(VIRQ_DOM_EXC);
 }
 
 /* Release resources belonging to task @p. */
diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index d6ae09b..a775aa3 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -994,6 +994,23 @@ long do_domctl(XEN_GUEST_HANDLE(xen_domctl_t) u_domctl)
     }
     break;
 
+    case XEN_DOMCTL_set_virq_handler:
+    {
+        struct domain *d;
+        int virq = op->u.set_virq_handler.virq;
+
+        ret = -ESRCH;
+        d = rcu_lock_domain_by_id(op->domain);
+        if ( d != NULL )
+        {
+            ret = xsm_set_virq_handler(d, virq);
+            if ( !ret )
+                ret = set_global_virq_handler(d, virq);
+            rcu_unlock_domain(d);
+        }
+    }
+    break;
+
     default:
         ret = arch_do_domctl(op, u_domctl);
         break;
diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c
index 9212042..77c7a27 100644
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -689,7 +689,7 @@ void send_guest_vcpu_virq(struct vcpu *v, int virq)
     spin_unlock_irqrestore(&v->virq_lock, flags);
 }
 
-void send_guest_global_virq(struct domain *d, int virq)
+static void send_guest_global_virq(struct domain *d, int virq)
 {
     unsigned long flags;
     int port;
@@ -739,6 +739,67 @@ int send_guest_pirq(struct domain *d, const struct pirq *pirq)
     return evtchn_set_pending(d->vcpu[chn->notify_vcpu_id], port);
 }
 
+static struct domain* global_virq_handlers[NR_VIRQS];
+
+spinlock_t global_virq_handlers_lock = SPIN_LOCK_UNLOCKED;
+
+static struct domain* _get_global_virq_handler(int virq)
+{
+    struct domain *d;
+
+    d = global_virq_handlers[virq];
+    return d != NULL ? d : dom0;
+}
+
+void send_global_virq(int virq)
+{
+    ASSERT(virq >= 0 && virq < NR_VIRQS);
+    ASSERT(virq_is_global(virq));
+
+    send_guest_global_virq(_get_global_virq_handler(virq), virq);
+}
+
+int set_global_virq_handler(struct domain *d, int virq)
+{
+    struct domain *old;
+
+    if (virq < 0 || virq >= NR_VIRQS)
+        return -EINVAL;
+    if (!virq_is_global(virq))
+        return -EINVAL;
+
+    if (global_virq_handlers[virq] == d)
+        return 0;
+
+    if (unlikely(!get_domain(d)))
+        return -EINVAL;
+
+    spin_lock(&global_virq_handlers_lock);
+
+    old = global_virq_handlers[virq];
+    global_virq_handlers[virq] = d;
+    if (old != NULL)
+        put_domain(old);
+    spin_unlock(&global_virq_handlers_lock);
+
+    return 0;
+}
+
+void clear_global_virq_handlers(struct domain *d)
+{
+    int virq;
+
+    spin_lock(&global_virq_handlers_lock);
+
+    for (virq = 0; virq < NR_VIRQS; virq++) {
+        if (global_virq_handlers[virq] == d) {
+            global_virq_handlers[virq] = NULL;
+            put_domain(d);
+        }
+    }
+
+    spin_unlock(&global_virq_handlers_lock);
+}
 
 static long evtchn_status(evtchn_status_t *status)
 {
diff --git a/xen/common/trace.c b/xen/common/trace.c
index 5772f24..58cbf39 100644
--- a/xen/common/trace.c
+++ b/xen/common/trace.c
@@ -661,7 +661,7 @@ static inline void insert_lost_records(struct t_buf *buf)
  */
 static void trace_notify_dom0(unsigned long unused)
 {
-    send_guest_global_virq(dom0, VIRQ_TBUF);
+    send_global_virq(VIRQ_TBUF);
 }
 static DECLARE_SOFTIRQ_TASKLET(trace_notify_dom0_tasklet,
                                trace_notify_dom0, 0);
diff --git a/xen/drivers/char/console.c b/xen/drivers/char/console.c
index 8a4c684..79b266f 100644
--- a/xen/drivers/char/console.c
+++ b/xen/drivers/char/console.c
@@ -287,7 +287,7 @@ static void __serial_rx(char c, struct cpu_user_regs *regs)
     if ( (serial_rx_prod-serial_rx_cons) != SERIAL_RX_SIZE )
         serial_rx_ring[SERIAL_RX_MASK(serial_rx_prod++)] = c;
     /* Always notify the guest: prevents receive path from getting stuck. */
-    send_guest_global_virq(dom0, VIRQ_CONSOLE);
+    send_global_virq(VIRQ_CONSOLE);
 }
 
 static void serial_rx(char c, struct cpu_user_regs *regs)
@@ -314,7 +314,7 @@ static void serial_rx(char c, struct cpu_user_regs *regs)
 
 static void notify_dom0_con_ring(unsigned long unused)
 {
-    send_guest_global_virq(dom0, VIRQ_CON_RING);
+    send_global_virq(VIRQ_CON_RING);
 }
 static DECLARE_SOFTIRQ_TASKLET(notify_dom0_con_ring_tasklet,
                                notify_dom0_con_ring, 0);
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index c7640aa..b1eb425 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -813,6 +813,12 @@ struct xen_domctl_audit_p2m {
 typedef struct xen_domctl_audit_p2m xen_domctl_audit_p2m_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_audit_p2m_t);
 
+struct xen_domctl_set_virq_handler {
+    uint32_t virq; /* IN */
+};
+typedef struct xen_domctl_set_virq_handler xen_domctl_set_virq_handler_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_virq_handler_t);
+
 #if defined(__i386__) || defined(__x86_64__)
 /* XEN_DOMCTL_setvcpuextstate */
 /* XEN_DOMCTL_getvcpuextstate */
@@ -912,6 +918,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_getvcpuextstate               63
 #define XEN_DOMCTL_set_access_required           64
 #define XEN_DOMCTL_audit_p2m                     65
+#define XEN_DOMCTL_set_virq_handler              71
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002
@@ -966,6 +973,7 @@ struct xen_domctl {
 #endif
         struct xen_domctl_set_access_required access_required;
         struct xen_domctl_audit_p2m         audit_p2m;
+        struct xen_domctl_set_virq_handler  set_virq_handler;
         struct xen_domctl_gdbsx_memio       gdbsx_guest_memio;
         struct xen_domctl_gdbsx_pauseunp_vcpu gdbsx_pauseunp_vcpu;
         struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
diff --git a/xen/include/xen/event.h b/xen/include/xen/event.h
index 7e5ad7b..2df65a0 100644
--- a/xen/include/xen/event.h
+++ b/xen/include/xen/event.h
@@ -24,11 +24,23 @@
 void send_guest_vcpu_virq(struct vcpu *v, int virq);
 
 /*
- * send_guest_global_virq: Notify guest via a global VIRQ.
- *  @d:        Domain to which virtual IRQ should be sent
+ * send_global_virq: Notify the domain handling a global VIRQ.
  *  @virq:     Virtual IRQ number (VIRQ_*)
  */
-void send_guest_global_virq(struct domain *d, int virq);
+void send_global_virq(int virq);
+
+/*
+ * sent_global_virq_handler: Set a global VIRQ handler.
+ *  @d:        New target domain for this VIRQ
+ *  @virq:     Virtual IRQ number (VIRQ_*), must be global
+ */
+int set_global_virq_handler(struct domain *d, int virq);
+
+/*
+ * clear_global_virq_handlers: Remove a domain as a handler for global VIRQs.
+ *  @d:        Domain to no longer handle global virtual IRQs
+ */
+void clear_global_virq_handlers(struct domain *d);
 
 /*
  * send_guest_pirq:
diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
index 566c808..c89c6ed 100644
--- a/xen/include/xsm/xsm.h
+++ b/xen/include/xsm/xsm.h
@@ -64,6 +64,7 @@ struct xsm_operations {
     int (*domain_settime) (struct domain *d);
     int (*set_target) (struct domain *d, struct domain *e);
     int (*domctl) (struct domain *d, int cmd);
+    int (*set_virq_handler) (struct domain *d, int virq);
     int (*tbufcontrol) (void);
     int (*readconsole) (uint32_t clear);
     int (*sched_id) (void);
@@ -265,6 +266,11 @@ static inline int xsm_domctl (struct domain *d, int cmd)
     return xsm_call(domctl(d, cmd));
 }
 
+static inline int xsm_set_virq_handler (struct domain *d, int virq)
+{
+    return xsm_call(set_virq_handler(d, virq));
+}
+
 static inline int xsm_tbufcontrol (void)
 {
     return xsm_call(tbufcontrol());
diff --git a/xen/xsm/dummy.c b/xen/xsm/dummy.c
index 65daa4e..59db86d 100644
--- a/xen/xsm/dummy.c
+++ b/xen/xsm/dummy.c
@@ -94,6 +94,11 @@ static int dummy_domctl(struct domain *d, int cmd)
     return 0;
 }
 
+static int dummy_set_virq_handler(struct domain *d, int virq)
+{
+    return 0;
+}
+
 static int dummy_tbufcontrol (void)
 {
     return 0;
@@ -596,6 +601,7 @@ void xsm_fixup_ops (struct xsm_operations *ops)
     set_to_dummy_if_null(ops, domain_settime);
     set_to_dummy_if_null(ops, set_target);
     set_to_dummy_if_null(ops, domctl);
+    set_to_dummy_if_null(ops, set_virq_handler);
     set_to_dummy_if_null(ops, tbufcontrol);
     set_to_dummy_if_null(ops, readconsole);
     set_to_dummy_if_null(ops, sched_id);
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index a2020a9..a1feb8d 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -597,6 +597,11 @@ static int flask_domctl(struct domain *d, int cmd)
     return domain_has_perm(current->domain, d, SECCLASS_DOMAIN, DOMAIN__SET_MISC_INFO);
 }
 
+static int flask_set_virq_handler(struct domain *d, int virq)
+{
+    return domain_has_perm(current->domain, d, SECCLASS_DOMAIN, DOMAIN__SET_VIRQ_HANDLER);
+}
+
 static int flask_tbufcontrol(void)
 {
     return domain_has_xen(current->domain, XEN__TBUFCONTROL);
@@ -1460,6 +1465,7 @@ static struct xsm_operations flask_ops = {
     .domain_settime = flask_domain_settime,
     .set_target = flask_set_target,
     .domctl = flask_domctl,
+    .set_virq_handler = flask_set_virq_handler,
     .tbufcontrol = flask_tbufcontrol,
     .readconsole = flask_readconsole,
     .sched_id = flask_sched_id,
diff --git a/xen/xsm/flask/include/av_perm_to_string.h b/xen/xsm/flask/include/av_perm_to_string.h
index 85cbffc..17a1c36 100644
--- a/xen/xsm/flask/include/av_perm_to_string.h
+++ b/xen/xsm/flask/include/av_perm_to_string.h
@@ -60,6 +60,7 @@
    S_(SECCLASS_DOMAIN, DOMAIN__GETPODTARGET, "getpodtarget")
    S_(SECCLASS_DOMAIN, DOMAIN__SETPODTARGET, "setpodtarget")
    S_(SECCLASS_DOMAIN, DOMAIN__SET_MISC_INFO, "set_misc_info")
+   S_(SECCLASS_DOMAIN, DOMAIN__SET_VIRQ_HANDLER, "set_virq_handler")
    S_(SECCLASS_HVM, HVM__SETHVMC, "sethvmc")
    S_(SECCLASS_HVM, HVM__GETHVMC, "gethvmc")
    S_(SECCLASS_HVM, HVM__SETPARAM, "setparam")
diff --git a/xen/xsm/flask/include/av_permissions.h b/xen/xsm/flask/include/av_permissions.h
index 9e55a86..42eaf81 100644
--- a/xen/xsm/flask/include/av_permissions.h
+++ b/xen/xsm/flask/include/av_permissions.h
@@ -61,6 +61,7 @@
 #define DOMAIN__GETPODTARGET                      0x10000000UL
 #define DOMAIN__SETPODTARGET                      0x20000000UL
 #define DOMAIN__SET_MISC_INFO                     0x40000000UL
+#define DOMAIN__SET_VIRQ_HANDLER                  0x80000000UL
 
 #define HVM__SETHVMC                              0x00000001UL
 #define HVM__GETHVMC                              0x00000002UL
-- 
1.7.7.5

Re: [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Jan Beulich]

>>> On 11.01.12 at 18:21, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> @@ -659,6 +659,8 @@ static void complete_domain_destroy(struct rcu_head *head)
>  
>      watchdog_domain_destroy(d);
>  
> +    clear_global_virq_handlers(d);

This is too late, I'm afraid. The domain can't possibly service the vIRQ(s)
anymore as soon as its destruction begins.

> +
>      rangeset_domain_destroy(d);
>  
>      cpupool_rm_domain(d);
> @@ -739,6 +739,67 @@ int send_guest_pirq(struct domain *d, const struct pirq *pirq)
>      return evtchn_set_pending(d->vcpu[chn->notify_vcpu_id], port);
>  }
>  
> +static struct domain* global_virq_handlers[NR_VIRQS];

__read_mostly?

Also, the formatting is wrong (also elsewhere in the patch) - the
space should be before the star, not after.

> +
> +spinlock_t global_virq_handlers_lock = SPIN_LOCK_UNLOCKED;

static? DEFINE_SPINLOCK()?

> +
> +static struct domain* _get_global_virq_handler(int virq)
> +{
> +    struct domain *d;
> +
> +    d = global_virq_handlers[virq];
> +    return d != NULL ? d : dom0;

This can be done in a single line:

    return global_virq_handlers[virq] ?: dom0;

> +}
> +
> +void send_global_virq(int virq)
> +{
> +    ASSERT(virq >= 0 && virq < NR_VIRQS);
> +    ASSERT(virq_is_global(virq));
> +
> +    send_guest_global_virq(_get_global_virq_handler(virq), virq);
> +}
> +
> +int set_global_virq_handler(struct domain *d, int virq)

In the domctl interface structure the virq (correctly) is uint32_t,
so why is it (signed) int here (and elsewhere)?

> +{
> +    struct domain *old;
> +
> +    if (virq < 0 || virq >= NR_VIRQS)

The < 0 part would then become unnecessary.

> +        return -EINVAL;
> +    if (!virq_is_global(virq))
> +        return -EINVAL;
> +
> +    if (global_virq_handlers[virq] == d)
> +        return 0;
> +
> +    if (unlikely(!get_domain(d)))
> +        return -EINVAL;
> +
> +    spin_lock(&global_virq_handlers_lock);
> +
> +    old = global_virq_handlers[virq];
> +    global_virq_handlers[virq] = d;
> +    if (old != NULL)
> +        put_domain(old);

This should happen outside the lock.

> +    spin_unlock(&global_virq_handlers_lock);
> +
> +    return 0;
> +}
> +
> +void clear_global_virq_handlers(struct domain *d)
> +{
> +    int virq;
> +
> +    spin_lock(&global_virq_handlers_lock);
> +
> +    for (virq = 0; virq < NR_VIRQS; virq++) {
> +        if (global_virq_handlers[virq] == d) {
> +            global_virq_handlers[virq] = NULL;
> +            put_domain(d);

Same here (albeit resulting in some code growth).

> +        }
> +    }
> +
> +    spin_unlock(&global_virq_handlers_lock);
> +}
>  
>  static long evtchn_status(evtchn_status_t *status)
>  {
> @@ -912,6 +918,7 @@ struct xen_domctl {
>  #define XEN_DOMCTL_getvcpuextstate               63
>  #define XEN_DOMCTL_set_access_required           64
>  #define XEN_DOMCTL_audit_p2m                     65
> +#define XEN_DOMCTL_set_virq_handler              71

Any reason for picking a non-contiguous value here?

>  #define XEN_DOMCTL_gdbsx_guestmemio            1000
>  #define XEN_DOMCTL_gdbsx_pausevcpu             1001
>  #define XEN_DOMCTL_gdbsx_unpausevcpu           1002

Jan

[PATCH 03/18] xsm: allow use of XEN_DOMCTL_getdomaininfo by non-IS_PRIV domains

[Daniel De Graaf]

This domctl does not allow manipulation of domains, only basic
information such as size and state. XSM modules can also provide
fine-grained control over what domains are visible to domains that call
getdomaininfo.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 xen/common/domctl.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index a775aa3..2c1ca85 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -263,6 +263,10 @@ long do_domctl(XEN_GUEST_HANDLE(xen_domctl_t) u_domctl)
             return -EPERM;
         break;
     }
+#ifdef XSM_ENABLE
+    case XEN_DOMCTL_getdomaininfo:
+        break;
+#endif
     default:
         if ( !IS_PRIV(current->domain) )
             return -EPERM;
-- 
1.7.7.5

Re: [PATCH 03/18] xsm: allow use of XEN_DOMCTL_getdomaininfo by non-IS_PRIV domains

[Keir Fraser]

On 11/01/2012 17:21, "Daniel De Graaf" <dgdegra@tycho.nsa.gov> wrote:

> This domctl does not allow manipulation of domains, only basic
> information such as size and state. XSM modules can also provide
> fine-grained control over what domains are visible to domains that call
> getdomaininfo.

Well there's a reason we might not disallow the hypercall. But why would we
actually care to allow it?

 -- Keir

> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  xen/common/domctl.c |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/xen/common/domctl.c b/xen/common/domctl.c
> index a775aa3..2c1ca85 100644
> --- a/xen/common/domctl.c
> +++ b/xen/common/domctl.c
> @@ -263,6 +263,10 @@ long do_domctl(XEN_GUEST_HANDLE(xen_domctl_t) u_domctl)
>              return -EPERM;
>          break;
>      }
> +#ifdef XSM_ENABLE
> +    case XEN_DOMCTL_getdomaininfo:
> +        break;
> +#endif
>      default:
>          if ( !IS_PRIV(current->domain) )
>              return -EPERM;

Re: [PATCH 03/18] xsm: allow use of XEN_DOMCTL_getdomaininfo by non-IS_PRIV domains

[Daniel De Graaf]

On 01/11/2012 12:27 PM, Keir Fraser wrote:
> On 11/01/2012 17:21, "Daniel De Graaf" <dgdegra@tycho.nsa.gov> wrote:
> 
>> This domctl does not allow manipulation of domains, only basic
>> information such as size and state. XSM modules can also provide
>> fine-grained control over what domains are visible to domains that call
>> getdomaininfo.
> 
> Well there's a reason we might not disallow the hypercall. But why would we
> actually care to allow it?
> 
>  -- Keir
> 

Xenstored needs to be able to call getdomaininfo to determine what domain(s)
have been shut down when it receives the DOM_EXC VIRQ. This is used both to
clean up references to the domain (event/grant) and to fire the @releaseDomain
watch so that other domains also clean up after the domain.

Other than this hypercall, there is no reason to make xenstored's domain
privileged (the VIRQ must be delegated by dom0).

>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>> ---
>>  xen/common/domctl.c |    4 ++++
>>  1 files changed, 4 insertions(+), 0 deletions(-)
>>
>> diff --git a/xen/common/domctl.c b/xen/common/domctl.c
>> index a775aa3..2c1ca85 100644
>> --- a/xen/common/domctl.c
>> +++ b/xen/common/domctl.c
>> @@ -263,6 +263,10 @@ long do_domctl(XEN_GUEST_HANDLE(xen_domctl_t) u_domctl)
>>              return -EPERM;
>>          break;
>>      }
>> +#ifdef XSM_ENABLE
>> +    case XEN_DOMCTL_getdomaininfo:
>> +        break;
>> +#endif
>>      default:
>>          if ( !IS_PRIV(current->domain) )
>>              return -EPERM;
> 

Re: [PATCH 03/18] xsm: allow use of XEN_DOMCTL_getdomaininfo by non-IS_PRIV domains

[Keir Fraser]

On 11/01/2012 17:27, "Keir Fraser" <keir@xen.org> wrote:

> On 11/01/2012 17:21, "Daniel De Graaf" <dgdegra@tycho.nsa.gov> wrote:
> 
>> This domctl does not allow manipulation of domains, only basic
>> information such as size and state. XSM modules can also provide
>> fine-grained control over what domains are visible to domains that call
>> getdomaininfo.
> 
> Well there's a reason we might not disallow the hypercall. But why would we
> actually care to allow it?

Ah, I've now seen patch 00/18, so this is for xenstore stubdom.

Also this applies only to the XSM-enabled case, and just allows you to get
as far as the finer-grained xsm_getdomaininfo() check. Somehow I got the
ifdef the wrong way round in my head!

Okay, makes a lot of sense. However, if the dummy xsm module is supposed to
behave very similarly to a !XSM_ENABLE build (which is what I personally
would expect), then I think dummy_getdomaininfo() should be changed to
return 0 only when IS_PRIV(current->domain).

This of course will require a 'proper' XSM setup to be able to use the
xenstore stubdom, but probably setting eg XSM/Flask should be a core part of
setting up such a hardened Xen host anyway.

 -- Keir

>  -- Keir
> 
>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>> ---
>>  xen/common/domctl.c |    4 ++++
>>  1 files changed, 4 insertions(+), 0 deletions(-)
>> 
>> diff --git a/xen/common/domctl.c b/xen/common/domctl.c
>> index a775aa3..2c1ca85 100644
>> --- a/xen/common/domctl.c
>> +++ b/xen/common/domctl.c
>> @@ -263,6 +263,10 @@ long do_domctl(XEN_GUEST_HANDLE(xen_domctl_t) u_domctl)
>>              return -EPERM;
>>          break;
>>      }
>> +#ifdef XSM_ENABLE
>> +    case XEN_DOMCTL_getdomaininfo:
>> +        break;
>> +#endif
>>      default:
>>          if ( !IS_PRIV(current->domain) )
>>              return -EPERM;
> 
> 

[PATCH 04/18] xen: Preserve reserved grant entries when switching versions

[Daniel De Graaf]

In order for the toolstack to use reserved grant table entries, the
grant table for a guest must be initialized prior to the guest's boot.
When the guest switches grant table versions (necessary if the guest is
using v2 grant tables, or on kexec if switching grant versions), these
initial grants will be cleared. Instead of clearing them, preserve
the grants across the type change.

Attempting to use or preserve v2-only features such as sub-page grants
will produce invalid v1 grant entries, which (while they will not work)
is not a problem since a guest can always produce such invalid entries.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 xen/common/grant_table.c |   38 +++++++++++++++++++++++++++++++-------
 1 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index 014734d..8d4a4cb 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -2106,6 +2106,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
     struct domain *d = current->domain;
     struct grant_table *gt = d->grant_table;
     struct active_grant_entry *act;
+    grant_entry_v1_t reserved_entries[GNTTAB_NR_RESERVED_ENTRIES];
     long res;
     int i;
 
@@ -2126,7 +2127,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
     /* (You need to change the version number for e.g. kexec.) */
     if ( gt->gt_version != 0 )
     {
-        for ( i = 0; i < nr_grant_entries(gt); i++ )
+        for ( i = GNTTAB_NR_RESERVED_ENTRIES; i < nr_grant_entries(gt); i++ )
         {
             act = &active_entry(gt, i);
             if ( act->pin != 0 )
@@ -2151,15 +2152,38 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
             goto out_unlock;
     }
 
+    /* Preserve the first 8 entries (toolstack reserved grants) */
+    if (gt->gt_version == 1) {
+        memcpy(reserved_entries, gt->shared_v1[0], sizeof(reserved_entries));
+    } else if (gt->gt_version == 2) {
+        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES && i < nr_grant_entries(gt); i++ )
+        {
+            reserved_entries[i].flags = shared_entry_v2(gt, i).hdr.flags;
+            reserved_entries[i].domid = shared_entry_v2(gt, i).hdr.domid;
+            reserved_entries[i].frame = shared_entry_v2(gt, i).full_page.frame;
+            reserved_entries[i].flags |= status_entry(gt, i);
+        }
+    }
+
     if ( op.version < 2 && gt->gt_version == 2 )
         gnttab_unpopulate_status_frames(d, gt);
 
-    if ( op.version != gt->gt_version )
-    {
-        /* Make sure there's no crud left over in the table from the
-           old version. */
-        for ( i = 0; i < nr_grant_frames(gt); i++ )
-            memset(gt->shared_raw[i], 0, PAGE_SIZE);
+    /* Make sure there's no crud left over in the table from the
+       old version. */
+    for ( i = 0; i < nr_grant_frames(gt); i++ )
+        memset(gt->shared_raw[i], 0, PAGE_SIZE);
+
+    /* Restore the first 8 entries (toolstack reserved grants) */
+    if (gt->gt_version != 0 && op.version == 1) {
+        memcpy(gt->shared_v1[0], reserved_entries, sizeof(reserved_entries));
+    } else if (gt->gt_version != 0 && op.version == 2) {
+        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES; i++ )
+        {
+            status_entry(gt, i) = reserved_entries[i].flags & (GTF_reading|GTF_writing);
+            shared_entry_v2(gt, i).hdr.flags = reserved_entries[i].flags & ~(GTF_reading|GTF_writing);
+            shared_entry_v2(gt, i).hdr.domid = reserved_entries[i].domid;
+            shared_entry_v2(gt, i).full_page.frame = reserved_entries[i].frame;
+        }
     }
 
     gt->gt_version = op.version;
-- 
1.7.7.5

Re: [PATCH 04/18] xen: Preserve reserved grant entries when switching versions

[Jan Beulich]

>>> On 11.01.12 at 18:21, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> --- a/xen/common/grant_table.c
> +++ b/xen/common/grant_table.c
> @@ -2106,6 +2106,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
>      struct domain *d = current->domain;
>      struct grant_table *gt = d->grant_table;
>      struct active_grant_entry *act;
> +    grant_entry_v1_t reserved_entries[GNTTAB_NR_RESERVED_ENTRIES];

How much stack space does this consume? (Or in other words, where
does GNTTAB_NR_RESERVED_ENTRIES get defined? It's not in this
patch, I can't find it in earlier patches, and it also doesn't exist in the
current staging tree.)

Jan

>      long res;
>      int i;
>  
> @@ -2126,7 +2127,7 @@ 
> gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
>      /* (You need to change the version number for e.g. kexec.) */
>      if ( gt->gt_version != 0 )
>      {
> -        for ( i = 0; i < nr_grant_entries(gt); i++ )
> +        for ( i = GNTTAB_NR_RESERVED_ENTRIES; i < nr_grant_entries(gt); i++ 
> )
>          {
>              act = &active_entry(gt, i);
>              if ( act->pin != 0 )
> @@ -2151,15 +2152,38 @@ 
> gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
>              goto out_unlock;
>      }
>  
> +    /* Preserve the first 8 entries (toolstack reserved grants) */
> +    if (gt->gt_version == 1) {
> +        memcpy(reserved_entries, gt->shared_v1[0], sizeof(reserved_entries));
> +    } else if (gt->gt_version == 2) {
> +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES && i < 
> nr_grant_entries(gt); i++ )
> +        {
> +            reserved_entries[i].flags = shared_entry_v2(gt, i).hdr.flags;
> +            reserved_entries[i].domid = shared_entry_v2(gt, i).hdr.domid;
> +            reserved_entries[i].frame = shared_entry_v2(gt, 
> i).full_page.frame;
> +            reserved_entries[i].flags |= status_entry(gt, i);
> +        }
> +    }
> +
>      if ( op.version < 2 && gt->gt_version == 2 )
>          gnttab_unpopulate_status_frames(d, gt);
>  
> -    if ( op.version != gt->gt_version )
> -    {
> -        /* Make sure there's no crud left over in the table from the
> -           old version. */
> -        for ( i = 0; i < nr_grant_frames(gt); i++ )
> -            memset(gt->shared_raw[i], 0, PAGE_SIZE);
> +    /* Make sure there's no crud left over in the table from the
> +       old version. */
> +    for ( i = 0; i < nr_grant_frames(gt); i++ )
> +        memset(gt->shared_raw[i], 0, PAGE_SIZE);
> +
> +    /* Restore the first 8 entries (toolstack reserved grants) */
> +    if (gt->gt_version != 0 && op.version == 1) {
> +        memcpy(gt->shared_v1[0], reserved_entries, sizeof(reserved_entries));
> +    } else if (gt->gt_version != 0 && op.version == 2) {
> +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES; i++ )
> +        {
> +            status_entry(gt, i) = reserved_entries[i].flags & 
> (GTF_reading|GTF_writing);
> +            shared_entry_v2(gt, i).hdr.flags = reserved_entries[i].flags & 
> ~(GTF_reading|GTF_writing);
> +            shared_entry_v2(gt, i).hdr.domid = reserved_entries[i].domid;
> +            shared_entry_v2(gt, i).full_page.frame = 
> reserved_entries[i].frame;
> +        }
>      }
>  
>      gt->gt_version = op.version;
> -- 
> 1.7.7.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com 
> http://lists.xensource.com/xen-devel 

Re: [PATCH 04/18] xen: Preserve reserved grant entries when switching versions

[Ian Campbell]

On Thu, 2012-01-12 at 08:53 +0000, Jan Beulich wrote:
> >>> On 11.01.12 at 18:21, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> > --- a/xen/common/grant_table.c
> > +++ b/xen/common/grant_table.c
> > @@ -2106,6 +2106,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
> >      struct domain *d = current->domain;
> >      struct grant_table *gt = d->grant_table;
> >      struct active_grant_entry *act;
> > +    grant_entry_v1_t reserved_entries[GNTTAB_NR_RESERVED_ENTRIES];
> 
> How much stack space does this consume? (Or in other words, where
> does GNTTAB_NR_RESERVED_ENTRIES get defined? It's not in this
> patch, I can't find it in earlier patches, and it also doesn't exist in the
> current staging tree.)

I could have sworn I'd seen this somewhere but I can't for the life of
me find it now :-/

One place I did find it is in the Linux gnttab driver, but that's not
canonical.

Ian.

> 
> Jan
> 
> >      long res;
> >      int i;
> >  
> > @@ -2126,7 +2127,7 @@ 
> > gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
> >      /* (You need to change the version number for e.g. kexec.) */
> >      if ( gt->gt_version != 0 )
> >      {
> > -        for ( i = 0; i < nr_grant_entries(gt); i++ )
> > +        for ( i = GNTTAB_NR_RESERVED_ENTRIES; i < nr_grant_entries(gt); i++ 
> > )
> >          {
> >              act = &active_entry(gt, i);
> >              if ( act->pin != 0 )
> > @@ -2151,15 +2152,38 @@ 
> > gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
> >              goto out_unlock;
> >      }
> >  
> > +    /* Preserve the first 8 entries (toolstack reserved grants) */
> > +    if (gt->gt_version == 1) {
> > +        memcpy(reserved_entries, gt->shared_v1[0], sizeof(reserved_entries));
> > +    } else if (gt->gt_version == 2) {
> > +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES && i < 
> > nr_grant_entries(gt); i++ )
> > +        {
> > +            reserved_entries[i].flags = shared_entry_v2(gt, i).hdr.flags;
> > +            reserved_entries[i].domid = shared_entry_v2(gt, i).hdr.domid;
> > +            reserved_entries[i].frame = shared_entry_v2(gt, 
> > i).full_page.frame;
> > +            reserved_entries[i].flags |= status_entry(gt, i);
> > +        }
> > +    }
> > +
> >      if ( op.version < 2 && gt->gt_version == 2 )
> >          gnttab_unpopulate_status_frames(d, gt);
> >  
> > -    if ( op.version != gt->gt_version )
> > -    {
> > -        /* Make sure there's no crud left over in the table from the
> > -           old version. */
> > -        for ( i = 0; i < nr_grant_frames(gt); i++ )
> > -            memset(gt->shared_raw[i], 0, PAGE_SIZE);
> > +    /* Make sure there's no crud left over in the table from the
> > +       old version. */
> > +    for ( i = 0; i < nr_grant_frames(gt); i++ )
> > +        memset(gt->shared_raw[i], 0, PAGE_SIZE);
> > +
> > +    /* Restore the first 8 entries (toolstack reserved grants) */
> > +    if (gt->gt_version != 0 && op.version == 1) {
> > +        memcpy(gt->shared_v1[0], reserved_entries, sizeof(reserved_entries));
> > +    } else if (gt->gt_version != 0 && op.version == 2) {
> > +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES; i++ )
> > +        {
> > +            status_entry(gt, i) = reserved_entries[i].flags & 
> > (GTF_reading|GTF_writing);
> > +            shared_entry_v2(gt, i).hdr.flags = reserved_entries[i].flags & 
> > ~(GTF_reading|GTF_writing);
> > +            shared_entry_v2(gt, i).hdr.domid = reserved_entries[i].domid;
> > +            shared_entry_v2(gt, i).full_page.frame = 
> > reserved_entries[i].frame;
> > +        }
> >      }
> >  
> >      gt->gt_version = op.version;
> > -- 
> > 1.7.7.5
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com 
> > http://lists.xensource.com/xen-devel 
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 04/18] xen: Preserve reserved grant entries when switching versions

[Ian Campbell]

On Thu, 2012-01-12 at 09:49 +0000, Ian Campbell wrote:
> On Thu, 2012-01-12 at 08:53 +0000, Jan Beulich wrote:
> > >>> On 11.01.12 at 18:21, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> > > --- a/xen/common/grant_table.c
> > > +++ b/xen/common/grant_table.c
> > > @@ -2106,6 +2106,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
> > >      struct domain *d = current->domain;
> > >      struct grant_table *gt = d->grant_table;
> > >      struct active_grant_entry *act;
> > > +    grant_entry_v1_t reserved_entries[GNTTAB_NR_RESERVED_ENTRIES];
> > 
> > How much stack space does this consume? (Or in other words, where
> > does GNTTAB_NR_RESERVED_ENTRIES get defined? It's not in this
> > patch, I can't find it in earlier patches, and it also doesn't exist in the
> > current staging tree.)
> 
> I could have sworn I'd seen this somewhere but I can't for the life of
> me find it now :-/

Found it in patch #6.

+/* External tools reserve first few grant table entries. */
+#define GNTTAB_NR_RESERVED_ENTRIES     8
+#define GNTTAB_RESERVED_CONSOLE        0
+#define GNTTAB_RESERVED_XENSTORE       1
+/* (the next 6 are reserved for future use) */

These patches need to be reordered to make sure each step builds.

Ian.

> 
> One place I did find it is in the Linux gnttab driver, but that's not
> canonical.
> 
> Ian.
> 
> > 
> > Jan
> > 
> > >      long res;
> > >      int i;
> > >  
> > > @@ -2126,7 +2127,7 @@ 
> > > gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
> > >      /* (You need to change the version number for e.g. kexec.) */
> > >      if ( gt->gt_version != 0 )
> > >      {
> > > -        for ( i = 0; i < nr_grant_entries(gt); i++ )
> > > +        for ( i = GNTTAB_NR_RESERVED_ENTRIES; i < nr_grant_entries(gt); i++ 
> > > )
> > >          {
> > >              act = &active_entry(gt, i);
> > >              if ( act->pin != 0 )
> > > @@ -2151,15 +2152,38 @@ 
> > > gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
> > >              goto out_unlock;
> > >      }
> > >  
> > > +    /* Preserve the first 8 entries (toolstack reserved grants) */
> > > +    if (gt->gt_version == 1) {
> > > +        memcpy(reserved_entries, gt->shared_v1[0], sizeof(reserved_entries));
> > > +    } else if (gt->gt_version == 2) {
> > > +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES && i < 
> > > nr_grant_entries(gt); i++ )
> > > +        {
> > > +            reserved_entries[i].flags = shared_entry_v2(gt, i).hdr.flags;
> > > +            reserved_entries[i].domid = shared_entry_v2(gt, i).hdr.domid;
> > > +            reserved_entries[i].frame = shared_entry_v2(gt, 
> > > i).full_page.frame;
> > > +            reserved_entries[i].flags |= status_entry(gt, i);
> > > +        }
> > > +    }
> > > +
> > >      if ( op.version < 2 && gt->gt_version == 2 )
> > >          gnttab_unpopulate_status_frames(d, gt);
> > >  
> > > -    if ( op.version != gt->gt_version )
> > > -    {
> > > -        /* Make sure there's no crud left over in the table from the
> > > -           old version. */
> > > -        for ( i = 0; i < nr_grant_frames(gt); i++ )
> > > -            memset(gt->shared_raw[i], 0, PAGE_SIZE);
> > > +    /* Make sure there's no crud left over in the table from the
> > > +       old version. */
> > > +    for ( i = 0; i < nr_grant_frames(gt); i++ )
> > > +        memset(gt->shared_raw[i], 0, PAGE_SIZE);
> > > +
> > > +    /* Restore the first 8 entries (toolstack reserved grants) */
> > > +    if (gt->gt_version != 0 && op.version == 1) {
> > > +        memcpy(gt->shared_v1[0], reserved_entries, sizeof(reserved_entries));
> > > +    } else if (gt->gt_version != 0 && op.version == 2) {
> > > +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES; i++ )
> > > +        {
> > > +            status_entry(gt, i) = reserved_entries[i].flags & 
> > > (GTF_reading|GTF_writing);
> > > +            shared_entry_v2(gt, i).hdr.flags = reserved_entries[i].flags & 
> > > ~(GTF_reading|GTF_writing);
> > > +            shared_entry_v2(gt, i).hdr.domid = reserved_entries[i].domid;
> > > +            shared_entry_v2(gt, i).full_page.frame = 
> > > reserved_entries[i].frame;
> > > +        }
> > >      }
> > >  
> > >      gt->gt_version = op.version;
> > > -- 
> > > 1.7.7.5
> > > 
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com 
> > > http://lists.xensource.com/xen-devel 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[PATCH 05/18] tools/libxl: Add xenstore and console backend domain IDs to config

[Daniel De Graaf]

This changes the event channels set up in the start_info structure of
newly built domains to be associated with a domain other than dom0,
to support xenstored or xenconsoled running in stub domains.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/libxl/libxl_dom.c     |    4 ++--
 tools/libxl/libxl_types.idl |    2 ++
 tools/libxl/xl_cmdimpl.c    |    6 ++++++
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index a4725fe..e12d687 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -104,8 +104,8 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
         xc_shadow_control(ctx->xch, domid, XEN_DOMCTL_SHADOW_OP_SET_ALLOCATION, NULL, 0, &shadow, 0, NULL);
     }
 
-    state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, 0);
-    state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, 0);
+    state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, info->xenstore_dom);
+    state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, info->console_dom);
     state->vm_generationid_addr = 0;
     return 0;
 }
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 574dec7..013f5c6 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -168,6 +168,8 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("video_memkb",     uint32),
     ("shadow_memkb",    uint32),
     ("disable_migrate", bool),
+    ("xenstore_dom",    uint32),
+    ("console_dom",     uint32),
     ("cpuid",           libxl_cpuid_policy_list),
     ("type",            libxl_domain_type),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 8c30de1..bb1bb5b 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -661,6 +661,12 @@ static void parse_config_data(const char *configfile_filename_report,
     if (!xlu_cfg_get_long (config, "maxvcpus", &l, 0))
         b_info->max_vcpus = l;
 
+    if (!xlu_cfg_get_long (config, "xenstore_dom", &l, 0))
+        b_info->xenstore_dom = l;
+
+    if (!xlu_cfg_get_long (config, "console_dom", &l, 0))
+        b_info->console_dom = l;
+
     if (!xlu_cfg_get_long (config, "memory", &l, 0)) {
         b_info->max_memkb = l * 1024;
         b_info->target_memkb = b_info->max_memkb;
-- 
1.7.7.5

[PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

This patch claims one reserved grant entry for the console and another
for the xenstore. It modifies the builder to fill in the grant table
entries for the console and the xenstore.

This has not been tested with any kind of migration.

Previous versions of this patch have been sent to xen-devel. See
http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00610.html
http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01491.html

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/libxc/xc_dom.h              |   13 +++
 tools/libxc/xc_dom_boot.c         |  164 +++++++++++++++++++++++++++++++++++++
 tools/libxc/xc_dom_compat_linux.c |    2 +
 tools/libxc/xc_domain_restore.c   |   17 ++++
 tools/libxc/xc_hvm_build.c        |    1 +
 tools/libxl/libxl_dom.c           |   17 ++++-
 tools/libxl/libxl_internal.h      |    2 +
 xen/include/public/grant_table.h  |    6 ++
 8 files changed, 220 insertions(+), 2 deletions(-)

diff --git a/tools/libxc/xc_dom.h b/tools/libxc/xc_dom.h
index e72f066..6c36403 100644
--- a/tools/libxc/xc_dom.h
+++ b/tools/libxc/xc_dom.h
@@ -106,7 +106,9 @@ struct xc_dom_image {
     /* misc xen domain config stuff */
     unsigned long flags;
     unsigned int console_evtchn;
+    unsigned int console_domid;
     unsigned int xenstore_evtchn;
+    unsigned int xenstore_domid;
     xen_pfn_t shared_info_mfn;
 
     xc_interface *xch;
@@ -200,6 +202,17 @@ void *xc_dom_boot_domU_map(struct xc_dom_image *dom, xen_pfn_t pfn,
                            xen_pfn_t count);
 int xc_dom_boot_image(struct xc_dom_image *dom);
 int xc_dom_compat_check(struct xc_dom_image *dom);
+int xc_dom_gnttab_init(struct xc_dom_image *dom);
+int xc_dom_gnttab_hvm_seed(xc_interface *xch, uint32_t domid,
+                           unsigned long console_gmfn,
+                           unsigned long xenstore_gmfn,
+                           uint32_t console_domid,
+                           uint32_t xenstore_domid);
+int xc_dom_gnttab_seed(xc_interface *xch, uint32_t domid,
+                       unsigned long console_gmfn,
+                       unsigned long xenstore_gmfn,
+                       uint32_t console_domid,
+                       uint32_t xenstore_domid);
 
 /* --- debugging bits ---------------------------------------------- */
 
diff --git a/tools/libxc/xc_dom_boot.c b/tools/libxc/xc_dom_boot.c
index 65f60df..1a55a6d 100644
--- a/tools/libxc/xc_dom_boot.c
+++ b/tools/libxc/xc_dom_boot.c
@@ -34,6 +34,7 @@
 #include "xg_private.h"
 #include "xc_dom.h"
 #include <xen/hvm/params.h>
+#include <xen/grant_table.h>
 
 /* ------------------------------------------------------------------------ */
 
@@ -275,6 +276,169 @@ int xc_dom_boot_image(struct xc_dom_image *dom)
     return rc;
 }
 
+static unsigned long xc_dom_gnttab_setup(xc_interface *xch, uint32_t domid)
+{
+    DECLARE_HYPERCALL;
+    gnttab_setup_table_t setup_table;
+    DECLARE_HYPERCALL_BUFFER(unsigned long, gmfnp);
+    int rc;
+	unsigned long gmfn;
+
+    gmfnp = xc_hypercall_buffer_alloc(xch, gmfnp, sizeof(*gmfnp));
+	if (gmfnp == NULL)
+		return -1;
+
+    setup_table.dom = domid;
+    setup_table.nr_frames = 1;
+    set_xen_guest_handle(setup_table.frame_list, gmfnp);
+    setup_table.status = 0;
+
+    hypercall.op = __HYPERVISOR_grant_table_op;
+    hypercall.arg[0] = GNTTABOP_setup_table;
+    hypercall.arg[1] = (unsigned long) &setup_table;
+    hypercall.arg[2] = 1;
+
+    rc = do_xen_hypercall(xch, &hypercall);
+	gmfn = *gmfnp;
+    xc_hypercall_buffer_free(xch, gmfnp);
+
+    if ( rc != 0 || setup_table.status != GNTST_okay )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to setup domU grant table "
+                     "[errno=%d, status=%" PRId16 "]\n",
+                     __FUNCTION__, rc != 0 ? errno : 0, setup_table.status);
+        return -1;
+    }
+
+    return gmfn;
+}
+
+int xc_dom_gnttab_seed(xc_interface *xch, uint32_t domid,
+                       unsigned long console_gmfn,
+                       unsigned long xenstore_gmfn,
+                       uint32_t console_domid,
+                       uint32_t xenstore_domid)
+{
+
+    unsigned long gnttab_gmfn;
+    grant_entry_v1_t *gnttab;
+
+    gnttab_gmfn = xc_dom_gnttab_setup(xch, domid);
+    if ( gnttab_gmfn == -1 )
+        return -1;
+
+    gnttab = xc_map_foreign_range(xch,
+                                  domid,
+                                  PAGE_SIZE,
+                                  PROT_READ|PROT_WRITE,
+                                  gnttab_gmfn);
+    if ( gnttab == NULL )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to map domU grant table "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    if ( domid != console_domid  && console_gmfn != -1)
+    {
+        gnttab[GNTTAB_RESERVED_CONSOLE].flags = GTF_permit_access;
+        gnttab[GNTTAB_RESERVED_CONSOLE].domid = console_domid;
+        gnttab[GNTTAB_RESERVED_CONSOLE].frame = console_gmfn;
+    }
+    if ( domid != xenstore_domid && xenstore_gmfn != -1)
+    {
+        gnttab[GNTTAB_RESERVED_XENSTORE].flags = GTF_permit_access;
+        gnttab[GNTTAB_RESERVED_XENSTORE].domid = xenstore_domid;
+        gnttab[GNTTAB_RESERVED_XENSTORE].frame = xenstore_gmfn;
+    }
+
+    if ( munmap(gnttab, PAGE_SIZE) == -1 )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to unmap domU grant table "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    return 0;
+}
+
+int xc_dom_gnttab_hvm_seed(xc_interface *xch, uint32_t domid,
+                           unsigned long console_gpfn,
+                           unsigned long xenstore_gpfn,
+                           uint32_t console_domid,
+                           uint32_t xenstore_domid)
+{
+#define SCRATCH_PFN_GNTTAB 0xFFFFE
+
+    int rc;
+    struct xen_add_to_physmap xatp = {
+        .domid = domid,
+        .space = XENMAPSPACE_grant_table,
+        .idx   = 0, /* TODO: what's this? */
+        .gpfn  = SCRATCH_PFN_GNTTAB
+    };
+    struct xen_remove_from_physmap xrfp = {
+    .domid = domid,
+    .gpfn  = SCRATCH_PFN_GNTTAB
+    };
+
+    rc = do_memory_op(xch, XENMEM_add_to_physmap, &xatp, sizeof(xatp));
+    if ( rc != 0 )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to add gnttab to physmap "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    rc = xc_dom_gnttab_seed(xch, domid,
+                            console_gpfn, xenstore_gpfn,
+                            console_domid, xenstore_domid);
+    if (rc != 0)
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to seed gnttab entries\n",
+                     __FUNCTION__);
+        (void) do_memory_op(xch, XENMEM_remove_from_physmap, &xrfp, sizeof(xrfp));
+        return -1;
+    }
+
+    rc = do_memory_op(xch, XENMEM_remove_from_physmap, &xrfp, sizeof(xrfp));
+    if (rc != 0)
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to remove gnttab from physmap "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    return 0;
+}
+
+int xc_dom_gnttab_init(struct xc_dom_image *dom)
+{
+    unsigned long console_gmfn;
+    unsigned long xenstore_gmfn;
+    int autotranslated;
+
+    autotranslated = xc_dom_feature_translated(dom);
+    console_gmfn = autotranslated ?
+           dom->console_pfn : xc_dom_p2m_host(dom, dom->console_pfn);
+    xenstore_gmfn = autotranslated ?
+           dom->xenstore_pfn : xc_dom_p2m_host(dom, dom->xenstore_pfn);
+
+    return xc_dom_gnttab_seed(dom->xch, dom->guest_domid,
+                              console_gmfn, xenstore_gmfn,
+                              dom->console_domid, dom->xenstore_domid);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxc/xc_dom_compat_linux.c b/tools/libxc/xc_dom_compat_linux.c
index 0e78842..2183a3b 100644
--- a/tools/libxc/xc_dom_compat_linux.c
+++ b/tools/libxc/xc_dom_compat_linux.c
@@ -62,6 +62,8 @@ static int xc_linux_build_internal(struct xc_dom_image *dom,
         goto out;
     if ( (rc = xc_dom_boot_image(dom)) != 0 )
         goto out;
+    if ( (rc = xc_dom_gnttab_init(dom)) != 0)
+        goto out;
 
     *console_mfn = xc_dom_p2m_host(dom, dom->console_pfn);
     *store_mfn = xc_dom_p2m_host(dom, dom->xenstore_pfn);
diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
index 3fda6f8..23619da 100644
--- a/tools/libxc/xc_domain_restore.c
+++ b/tools/libxc/xc_domain_restore.c
@@ -2018,6 +2018,15 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
         memcpy(ctx->live_p2m, ctx->p2m, dinfo->p2m_size * sizeof(xen_pfn_t));
     munmap(ctx->live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);
 
+/* TODO don't hardcode zero here */
+    rc = xc_dom_gnttab_seed(xch, dom,
+                            *console_mfn, *store_mfn, 0, 0);
+    if (rc != 0)
+    {
+        ERROR("error seeding grant table");
+        goto out;
+    }
+
     DPRINTF("Domain ready to be built.\n");
     rc = 0;
     goto out;
@@ -2076,6 +2085,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
         goto out;
     }
 
+	/* TODO don't hardcode zero here */
+    rc = xc_dom_gnttab_hvm_seed(xch, dom, *console_mfn, *store_mfn, 0, 0);
+    if (rc != 0)
+    {
+        ERROR("error seeding grant table");
+        goto out;
+    }
+
     /* HVM success! */
     rc = 0;
 
diff --git a/tools/libxc/xc_hvm_build.c b/tools/libxc/xc_hvm_build.c
index 9831bab..453f8a5 100644
--- a/tools/libxc/xc_hvm_build.c
+++ b/tools/libxc/xc_hvm_build.c
@@ -24,6 +24,7 @@
 
 #include "xg_private.h"
 #include "xc_private.h"
+#include "xc_dom.h"
 
 #include <xen/foreign/x86_32.h>
 #include <xen/foreign/x86_64.h>
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index e12d687..7f7bca6 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -104,7 +104,9 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
         xc_shadow_control(ctx->xch, domid, XEN_DOMCTL_SHADOW_OP_SET_ALLOCATION, NULL, 0, &shadow, 0, NULL);
     }
 
+    state->store_domid = info->xenstore_dom;
     state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, info->xenstore_dom);
+    state->console_domid = info->console_dom;
     state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, info->console_dom);
     state->vm_generationid_addr = 0;
     return 0;
@@ -223,7 +225,9 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
 
     dom->flags = flags;
     dom->console_evtchn = state->console_port;
+    dom->console_domid = state->console_domid;
     dom->xenstore_evtchn = state->store_port;
+    dom->xenstore_domid = state->store_domid;
 
     if ( (ret = xc_dom_boot_xen_init(dom, ctx->xch, domid)) != 0 ) {
         LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "xc_dom_boot_xen_init failed");
@@ -249,6 +253,10 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
         LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "xc_dom_boot_image failed");
         goto out;
     }
+    if ( (ret = xc_dom_gnttab_init(dom)) != 0 ) {
+        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "xc_dom_gnttab_init failed");
+        goto out;
+    }
 
     state->console_mfn = xc_dom_p2m_host(dom, dom->console_pfn);
     state->store_mfn = xc_dom_p2m_host(dom, dom->xenstore_pfn);
@@ -262,7 +270,8 @@ out:
 static int hvm_build_set_params(xc_interface *handle, uint32_t domid,
                                 libxl_domain_build_info *info,
                                 int store_evtchn, unsigned long *store_mfn,
-                                int console_evtchn, unsigned long *console_mfn)
+                                int console_evtchn, unsigned long *console_mfn,
+                                uint32_t store_domid, uint32_t console_domid)
 {
     struct hvm_info_table *va_hvm;
     uint8_t *va_map, sum;
@@ -295,6 +304,8 @@ static int hvm_build_set_params(xc_interface *handle, uint32_t domid,
     xc_set_hvm_param(handle, domid, HVM_PARAM_NESTEDHVM, info->u.hvm.nested_hvm);
     xc_set_hvm_param(handle, domid, HVM_PARAM_STORE_EVTCHN, store_evtchn);
     xc_set_hvm_param(handle, domid, HVM_PARAM_CONSOLE_EVTCHN, console_evtchn);
+
+    xc_dom_gnttab_hvm_seed(handle, domid, *console_mfn, *store_mfn, console_domid, store_domid);
     return 0;
 }
 
@@ -348,7 +359,9 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
         goto out;
     }
     ret = hvm_build_set_params(ctx->xch, domid, info, state->store_port,
-                               &state->store_mfn, state->console_port, &state->console_mfn);
+                               &state->store_mfn, state->console_port,
+                               &state->console_mfn, state->store_domid,
+                               state->console_domid);
     if (ret) {
         LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, ret, "hvm build set params failed");
         goto out;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 288c03c..97ead08 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -219,9 +219,11 @@ _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
     libxl__domain_type((gc), (domid)) == LIBXL_DOMAIN_TYPE_##type
 typedef struct {
     uint32_t store_port;
+    uint32_t store_domid;
     unsigned long store_mfn;
 
     uint32_t console_port;
+    uint32_t console_domid;
     unsigned long console_mfn;
     unsigned long vm_generationid_addr;
 } libxl__domain_build_state;
diff --git a/xen/include/public/grant_table.h b/xen/include/public/grant_table.h
index 0bf20bc..36d1ac7 100644
--- a/xen/include/public/grant_table.h
+++ b/xen/include/public/grant_table.h
@@ -117,6 +117,12 @@ struct grant_entry_v1 {
 };
 typedef struct grant_entry_v1 grant_entry_v1_t;
 
+/* External tools reserve first few grant table entries. */
+#define GNTTAB_NR_RESERVED_ENTRIES     8
+#define GNTTAB_RESERVED_CONSOLE        0
+#define GNTTAB_RESERVED_XENSTORE       1
+/* (the next 6 are reserved for future use) */
+
 /*
  * Type of grant entry.
  *  GTF_invalid: This grant entry grants no privileges.
-- 
1.7.7.5

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Ian Campbell]

On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> 
> This patch claims one reserved grant entry for the console and another
> for the xenstore. It modifies the builder to fill in the grant table
> entries for the console and the xenstore.
> 
> This has not been tested with any kind of migration.
> 
[...]
> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
> index 3fda6f8..23619da 100644
> --- a/tools/libxc/xc_domain_restore.c
> +++ b/tools/libxc/xc_domain_restore.c
> @@ -2018,6 +2018,15 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>          memcpy(ctx->live_p2m, ctx->p2m, dinfo->p2m_size * sizeof(xen_pfn_t));
>      munmap(ctx->live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);
> 
> +/* TODO don't hardcode zero here */
> +    rc = xc_dom_gnttab_seed(xch, dom,
> +                            *console_mfn, *store_mfn, 0, 0);

Presumably this TODO is the source of the comment in the changelog WRT
migration.

Does it Just Work or is there a legitimate TODO item here?

Ian.

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Daniel De Graaf]

On 01/12/2012 04:59 AM, Ian Campbell wrote:
> On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
>> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
>>
>> This patch claims one reserved grant entry for the console and another
>> for the xenstore. It modifies the builder to fill in the grant table
>> entries for the console and the xenstore.
>>
>> This has not been tested with any kind of migration.
>>
> [...]
>> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
>> index 3fda6f8..23619da 100644
>> --- a/tools/libxc/xc_domain_restore.c
>> +++ b/tools/libxc/xc_domain_restore.c
>> @@ -2018,6 +2018,15 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>>          memcpy(ctx->live_p2m, ctx->p2m, dinfo->p2m_size * sizeof(xen_pfn_t));
>>      munmap(ctx->live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);
>>
>> +/* TODO don't hardcode zero here */
>> +    rc = xc_dom_gnttab_seed(xch, dom,
>> +                            *console_mfn, *store_mfn, 0, 0);
> 
> Presumably this TODO is the source of the comment in the changelog WRT
> migration.
> 
> Does it Just Work or is there a legitimate TODO item here?
> 
> Ian.
> 
> 

This causes migration to only work if xenstored/xenconsoled are both in
dom0, as the domain ID for both of them are hardcoded to zero. Determining
the correct values for these domain IDs is more difficult than the domain
build case, because they may not be the same as when the domain was built
(especially if we are migrating).

The previous patch series used a domid file named similarly to a pid file
to identify the location of xenstored and xenconsoled; this method would
allow the TODO to be resolved. I think a better solution is to refer to the
xenstore/xenconsole domains by name instead of domid, and set the names in
a configuration file (/etc/xen/xl.conf?). This would also replace the
xenstore_dom/console_dom parameters in patch #5.

-- 
Daniel De Graaf
National Security Agency

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Ian Campbell]

On Thu, 2012-01-12 at 15:11 +0000, Daniel De Graaf wrote:
> On 01/12/2012 04:59 AM, Ian Campbell wrote:
> > On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> >> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> >>
> >> This patch claims one reserved grant entry for the console and another
> >> for the xenstore. It modifies the builder to fill in the grant table
> >> entries for the console and the xenstore.
> >>
> >> This has not been tested with any kind of migration.
> >>
> > [...]
> >> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
> >> index 3fda6f8..23619da 100644
> >> --- a/tools/libxc/xc_domain_restore.c
> >> +++ b/tools/libxc/xc_domain_restore.c
> >> @@ -2018,6 +2018,15 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
> >>          memcpy(ctx->live_p2m, ctx->p2m, dinfo->p2m_size * sizeof(xen_pfn_t));
> >>      munmap(ctx->live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);
> >>
> >> +/* TODO don't hardcode zero here */
> >> +    rc = xc_dom_gnttab_seed(xch, dom,
> >> +                            *console_mfn, *store_mfn, 0, 0);
> > 
> > Presumably this TODO is the source of the comment in the changelog WRT
> > migration.
> > 
> > Does it Just Work or is there a legitimate TODO item here?
> > 
> > Ian.
> > 
> > 
> 
> This causes migration to only work if xenstored/xenconsoled are both in
> dom0, as the domain ID for both of them are hardcoded to zero. Determining
> the correct values for these domain IDs is more difficult than the domain
> build case, because they may not be the same as when the domain was built
> (especially if we are migrating).
> 
> The previous patch series used a domid file named similarly to a pid file
> to identify the location of xenstored and xenconsoled; this method would
> allow the TODO to be resolved. I think a better solution is to refer to the
> xenstore/xenconsole domains by name instead of domid, and set the names in
> a configuration file (/etc/xen/xl.conf?). This would also replace the
> xenstore_dom/console_dom parameters in patch #5.

That would work. You could also stash the necessary parameters in
xenstore from the tool which starts the stub-xenstored such that the
toolstack can look them up as needed. That avoids having to have an
xl.conf variable.

Ian.

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Ian Jackson]

Daniel De Graaf writes ("Re: [Xen-devel] [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants"):
> The previous patch series used a domid file named similarly to a pid file
> to identify the location of xenstored and xenconsoled; this method would
> allow the TODO to be resolved. I think a better solution is to refer to the
> xenstore/xenconsole domains by name instead of domid, and set the names in
> a configuration file (/etc/xen/xl.conf?). This would also replace the
> xenstore_dom/console_dom parameters in patch #5.

Clearly this isn't a configuration parameter; it's a runtime value.
You don't store pids in config files.

If this can't go in xenstore itself then a file in /var/run/xen would
do just fine.

Ian.

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Daniel De Graaf]

On 01/12/2012 12:21 PM, Ian Jackson wrote:
> Daniel De Graaf writes ("Re: [Xen-devel] [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants"):
>> The previous patch series used a domid file named similarly to a pid file
>> to identify the location of xenstored and xenconsoled; this method would
>> allow the TODO to be resolved. I think a better solution is to refer to the
>> xenstore/xenconsole domains by name instead of domid, and set the names in
>> a configuration file (/etc/xen/xl.conf?). This would also replace the
>> xenstore_dom/console_dom parameters in patch #5.
> 
> Clearly this isn't a configuration parameter; it's a runtime value.
> You don't store pids in config files.

I was suggesting storing the domain name in the config file, which wouldn't
be a runtime value. However, I think it should be possible to place this in
xenstore, and that will probably be better as that's where you would expect
things like this to be.

> If this can't go in xenstore itself then a file in /var/run/xen would
> do just fine.
> 
> Ian.
> 

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Ian Jackson]

Daniel De Graaf writes ("Re: [Xen-devel] [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants"):
> I was suggesting storing the domain name in the config file, which wouldn't
> be a runtime value. However, I think it should be possible to place this in
> xenstore, and that will probably be better as that's where you would expect
> things like this to be.

The registry of domain names is in xenstore, so that's not going to
work :-).

Ian.

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Ian Campbell]

On Thu, 2012-01-12 at 17:35 +0000, Ian Jackson wrote:
> Daniel De Graaf writes ("Re: [Xen-devel] [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants"):
> > I was suggesting storing the domain name in the config file, which wouldn't
> > be a runtime value. However, I think it should be possible to place this in
> > xenstore, and that will probably be better as that's where you would expect
> > things like this to be.
> 
> The registry of domain names is in xenstore, so that's not going to
> work :-).

This info is used for starting subsequent domains not the xenstore
domain itself so I think it will.

Whichever process starts the xenstore stubdom watis for it to initialise
and then writes the necessary parameters to xenstore. Future toolstack
invocations can then read it back. The toolstack doesn't need to know
where the xenstore is running since it just uses /proc/xen/xenbus.

Ian.

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Daniel De Graaf]

On 01/12/2012 12:35 PM, Ian Jackson wrote:
> Daniel De Graaf writes ("Re: [Xen-devel] [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants"):
>> I was suggesting storing the domain name in the config file, which wouldn't
>> be a runtime value. However, I think it should be possible to place this in
>> xenstore, and that will probably be better as that's where you would expect
>> things like this to be.
> 
> The registry of domain names is in xenstore, so that's not going to
> work :-).
> 
> Ian.
> 

Actually, it'll work just fine: dom0 has a connection to xenstore, it just
can't (from userspace) tell what domid it's talking to. It's really the same
thing as just storing the xenstored domid in xenstore; /tool/xenstored/domid
is where I plan to put it.

-- 
Daniel De Graaf
National Security Agency

[PATCH 07/18] mini-os: avoid crash if no console is provided

[Daniel De Graaf]

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/console/xencons_ring.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/extras/mini-os/console/xencons_ring.c b/extras/mini-os/console/xencons_ring.c
index 22fd618..14a8bd1 100644
--- a/extras/mini-os/console/xencons_ring.c
+++ b/extras/mini-os/console/xencons_ring.c
@@ -25,7 +25,10 @@ static inline void notify_daemon(struct consfront_dev *dev)
 
 static inline struct xencons_interface *xencons_interface(void)
 {
-    return mfn_to_virt(start_info.console.domU.mfn);
+    if (start_info.console.domU.evtchn)
+        return mfn_to_virt(start_info.console.domU.mfn);
+    else
+        return NULL;
 } 
  
 int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, unsigned len)
@@ -38,6 +41,8 @@ int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, uns
             intf = xencons_interface();
         else
             intf = dev->ring;
+    if (!intf)
+        return sent;
 
 	cons = intf->out_cons;
 	prod = intf->out_prod;
-- 
1.7.7.5

Re: [PATCH 07/18] mini-os: avoid crash if no console is provided

[Ian Campbell]

On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:

When/why does this happen? 

I guess it is because when starting the xenstore domain you cannot use
xenstore to communicate with xenconsoled (and/or it isn't even running
yet).

I wonder if there is a way we can do lazy-setup of the console for just
the xenstore domain?

Alternatively I seem to recall a little tool which Diego wrote to pull
the console ring out of a domain directly as a debuging aid but that
relies on us setting up a console ring and evtchn even if xenconsoled
cannot be involved which makes this patch unnecessary.

Ian.

> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  extras/mini-os/console/xencons_ring.c |    7 ++++++-
>  1 files changed, 6 insertions(+), 1 deletions(-)
> 
> diff --git a/extras/mini-os/console/xencons_ring.c b/extras/mini-os/console/xencons_ring.c
> index 22fd618..14a8bd1 100644
> --- a/extras/mini-os/console/xencons_ring.c
> +++ b/extras/mini-os/console/xencons_ring.c
> @@ -25,7 +25,10 @@ static inline void notify_daemon(struct consfront_dev *dev)
>  
>  static inline struct xencons_interface *xencons_interface(void)
>  {
> -    return mfn_to_virt(start_info.console.domU.mfn);
> +    if (start_info.console.domU.evtchn)
> +        return mfn_to_virt(start_info.console.domU.mfn);
> +    else
> +        return NULL;
>  } 
>   
>  int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, unsigned len)
> @@ -38,6 +41,8 @@ int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, uns
>              intf = xencons_interface();
>          else
>              intf = dev->ring;
> +    if (!intf)
> +        return sent;
>  
>  	cons = intf->out_cons;
>  	prod = intf->out_prod;

Re: [PATCH 07/18] mini-os: avoid crash if no console is provided

[Daniel De Graaf]

On 01/12/2012 05:03 AM, Ian Campbell wrote:
> On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> 
> When/why does this happen? 

Initially, when I use a custom domain builder to create the xenstored
domain without a xenconsoled running yet (as xenconsoled needs xenstore).

> I guess it is because when starting the xenstore domain you cannot use
> xenstore to communicate with xenconsoled (and/or it isn't even running
> yet).
> 
> I wonder if there is a way we can do lazy-setup of the console for just
> the xenstore domain?

Xenstored will produce output on the hypervisor console if Xen is compiled
with debugging. If xenstored produces a lot of output, it can block on
waiting for xenconsoled, which might be blocking on xenstored - so I don't
think we want to hook it up to the console in the normal case, or at least
not the same xenconsoled that talks to domUs.
 
> Alternatively I seem to recall a little tool which Diego wrote to pull
> the console ring out of a domain directly as a debuging aid but that
> relies on us setting up a console ring and evtchn even if xenconsoled
> cannot be involved which makes this patch unnecessary.
> 
> Ian.

This runs into the same blocking problem if xenstored produces too much
output.

Re: [PATCH 07/18] mini-os: avoid crash if no console is provided

[Ian Campbell]

On Thu, 2012-01-12 at 17:56 +0000, Daniel De Graaf wrote:
> On 01/12/2012 05:03 AM, Ian Campbell wrote:
> > On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> > 
> > When/why does this happen? 
> 
> Initially, when I use a custom domain builder to create the xenstored
> domain without a xenconsoled running yet (as xenconsoled needs xenstore).

Right, that makes sense.

> > I guess it is because when starting the xenstore domain you cannot use
> > xenstore to communicate with xenconsoled (and/or it isn't even running
> > yet).
> > 
> > I wonder if there is a way we can do lazy-setup of the console for just
> > the xenstore domain?
> 
> Xenstored will produce output on the hypervisor console if Xen is compiled
> with debugging. If xenstored produces a lot of output, it can block on
> waiting for xenconsoled, which might be blocking on xenstored - so I don't
> think we want to hook it up to the console in the normal case, or at least
> not the same xenconsoled that talks to domUs.

Hrm, yes.
 
> > Alternatively I seem to recall a little tool which Diego wrote to pull
> > the console ring out of a domain directly as a debuging aid but that
> > relies on us setting up a console ring and evtchn even if xenconsoled
> > cannot be involved which makes this patch unnecessary.

> This runs into the same blocking problem if xenstored produces too much
> output.

What about if the tools/xenstore/init-xenstore-domain tool you posted in
your v2 provides console_mfn and console_evtch to the stubdom (via the
shared info) and after it has started the domain it would daemonize and
sit there pumping stuff out of the stubdom console ring and
into /var/log or syslog or something. There should be no interaction
with xenstore there, should there?

Ian.

[PATCH 08/18] mini-os: avoid crash if no xenstore is provided

[Daniel De Graaf]

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/xenbus/xenbus.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/extras/mini-os/xenbus/xenbus.c b/extras/mini-os/xenbus/xenbus.c
index a8081fd..e475e2c 100644
--- a/extras/mini-os/xenbus/xenbus.c
+++ b/extras/mini-os/xenbus/xenbus.c
@@ -328,6 +328,10 @@ static int allocate_xenbus_id(void)
 void init_xenbus(void)
 {
     int err;
+    if (!start_info.store_evtchn) {
+        printk("Skipping initialization of xenbus\n");
+        return;
+    }
     printk("Initialising xenbus\n");
     DEBUG("init_xenbus called.\n");
     xenstore_buf = mfn_to_virt(start_info.store_mfn);
@@ -435,6 +439,11 @@ xenbus_msg_reply(int type,
     DEFINE_WAIT(w);
     struct xsd_sockmsg *rep;
 
+    if (!xenstore_buf) {
+        printk("xenbus_msg_reply called but no xenstore!\n");
+        return NULL;
+    }
+
     id = allocate_xenbus_id();
     add_waiter(w, req_info[id].waitq);
 
-- 
1.7.7.5

[PATCH 09/18] mini-os: remove per-fd evtchn limit

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

Changes the minios evtchn implementation to use a list instead of an array.
This allows it to grow as necessary to support any number of ports.
Unfortunately, it's still limited by NR_EVS in events.c.

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/include/lib.h |   16 +++---
 tools/libxc/xc_minios.c      |  139 ++++++++++++++++++++++--------------------
 2 files changed, 81 insertions(+), 74 deletions(-)

diff --git a/extras/mini-os/include/lib.h b/extras/mini-os/include/lib.h
index bd3eeaf..12070c3 100644
--- a/extras/mini-os/include/lib.h
+++ b/extras/mini-os/include/lib.h
@@ -53,6 +53,7 @@
 #include <xen/xen.h>
 #include <xen/event_channel.h>
 #include "gntmap.h"
+#include "list.h"
 
 #ifdef HAVE_LIBC
 #include <stdio.h>
@@ -143,7 +144,12 @@ enum fd_type {
     FTYPE_SAVEFILE,
 };
 
-#define MAX_EVTCHN_PORTS 16
+struct evtchn_port_info {
+        struct minios_list_head list;
+        evtchn_port_t port;
+        unsigned long pending;
+        int bound;
+};
 
 extern struct file {
     enum fd_type type;
@@ -158,13 +164,7 @@ extern struct file {
 	    off_t offset;
 	} file;
 	struct {
-            /* To each event channel FD is associated a series of ports which
-             * wakes select for this FD. */
-            struct {
-                evtchn_port_t port;
-                unsigned long pending;
-                int bound;
-            } ports[MAX_EVTCHN_PORTS];
+	    struct minios_list_head ports;
 	} evtchn;
 	struct gntmap gntmap;
 	struct {
diff --git a/tools/libxc/xc_minios.c b/tools/libxc/xc_minios.c
index 8bbfd18..29cce63 100644
--- a/tools/libxc/xc_minios.c
+++ b/tools/libxc/xc_minios.c
@@ -210,15 +210,34 @@ static struct xc_osdep_ops minios_privcmd_ops = {
     },
 };
 
+
+/* XXX Note: This is not threadsafe */
+static struct evtchn_port_info* port_alloc(int fd) {
+    struct evtchn_port_info *port_info;
+    port_info = malloc(sizeof(struct evtchn_port_info));
+    if (port_info == NULL)
+        return NULL;
+    port_info->pending = 0;
+    port_info->port = -1;
+    port_info->bound = 0;
+
+    minios_list_add(&port_info->list, &files[fd].evtchn.ports);
+    return port_info;
+}
+
+static void port_dealloc(struct evtchn_port_info *port_info) {
+    if (port_info->bound)
+        unbind_evtchn(port_info->port);
+    minios_list_del(&port_info->list);
+    free(port_info);
+}
+
 static xc_osdep_handle minios_evtchn_open(xc_evtchn *xce)
 {
-    int fd = alloc_fd(FTYPE_EVTCHN), i;
+    int fd = alloc_fd(FTYPE_EVTCHN);
     if ( fd == -1 )
         return XC_OSDEP_OPEN_ERROR;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++) {
-	files[fd].evtchn.ports[i].port = -1;
-        files[fd].evtchn.ports[i].bound = 0;
-    }
+    MINIOS_INIT_LIST_HEAD(&files[fd].evtchn.ports);
     printf("evtchn_open() -> %d\n", fd);
     return (xc_osdep_handle)fd;
 }
@@ -231,10 +250,10 @@ static int minios_evtchn_close(xc_evtchn *xce, xc_osdep_handle h)
 
 void minios_evtchn_close_fd(int fd)
 {
-    int i;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++)
-        if (files[fd].evtchn.ports[i].bound)
-            unbind_evtchn(files[fd].evtchn.ports[i].port);
+    struct evtchn_port_info *port_info, *tmp;
+    minios_list_for_each_entry_safe(port_info, tmp, &files[fd].evtchn.ports, list)
+        port_dealloc(port_info);
+
     files[fd].type = FTYPE_NONE;
 }
 
@@ -256,35 +275,21 @@ static int minios_evtchn_notify(xc_evtchn *xce, xc_osdep_handle h, evtchn_port_t
     return ret;
 }
 
-/* XXX Note: This is not threadsafe */
-static int port_alloc(int fd) {
-    int i;
-    for (i= 0; i < MAX_EVTCHN_PORTS; i++)
-	if (files[fd].evtchn.ports[i].port == -1)
-	    break;
-    if (i == MAX_EVTCHN_PORTS) {
-	printf("Too many ports in xc handle\n");
-	errno = EMFILE;
-	return -1;
-    }
-    files[fd].evtchn.ports[i].pending = 0;
-    return i;
-}
-
 static void evtchn_handler(evtchn_port_t port, struct pt_regs *regs, void *data)
 {
     int fd = (int)(intptr_t)data;
-    int i;
+    struct evtchn_port_info *port_info;
     assert(files[fd].type == FTYPE_EVTCHN);
     mask_evtchn(port);
-    for (i= 0; i < MAX_EVTCHN_PORTS; i++)
-	if (files[fd].evtchn.ports[i].port == port)
-	    break;
-    if (i == MAX_EVTCHN_PORTS) {
-	printk("Unknown port for handle %d\n", fd);
-	return;
+    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
+        if (port_info->port == port)
+            goto found;
     }
-    files[fd].evtchn.ports[i].pending = 1;
+    printk("Unknown port for handle %d\n", fd);
+    return;
+
+ found:
+    port_info->pending = 1;
     files[fd].read = 1;
     wake_up(&event_queue);
 }
@@ -292,12 +297,13 @@ static void evtchn_handler(evtchn_port_t port, struct pt_regs *regs, void *data)
 static evtchn_port_or_error_t minios_evtchn_bind_unbound_port(xc_evtchn *xce, xc_osdep_handle h, int domid)
 {
     int fd = (int)h;
-    int ret, i;
+    struct evtchn_port_info *port_info;
+    int ret;
     evtchn_port_t port;
 
     assert(get_current() == main_thread);
-    i = port_alloc(fd);
-    if (i == -1)
+    port_info = port_alloc(fd);
+    if (port_info == NULL)
 	return -1;
 
     printf("xc_evtchn_bind_unbound_port(%d)", domid);
@@ -305,11 +311,12 @@ static evtchn_port_or_error_t minios_evtchn_bind_unbound_port(xc_evtchn *xce, xc
     printf(" = %d\n", ret);
 
     if (ret < 0) {
+	port_dealloc(port_info);
 	errno = -ret;
 	return -1;
     }
-    files[fd].evtchn.ports[i].bound = 1;
-    files[fd].evtchn.ports[i].port = port;
+    port_info->bound = 1;
+    port_info->port = port;
     unmask_evtchn(port);
     return port;
 }
@@ -318,12 +325,13 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
     evtchn_port_t remote_port)
 {
     int fd = (int)h;
+    struct evtchn_port_info *port_info;
     evtchn_port_t local_port;
-    int ret, i;
+    int ret;
 
     assert(get_current() == main_thread);
-    i = port_alloc(fd);
-    if (i == -1)
+    port_info = port_alloc(fd);
+    if (port_info == NULL)
 	return -1;
 
     printf("xc_evtchn_bind_interdomain(%d, %"PRId32")", domid, remote_port);
@@ -331,11 +339,12 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
     printf(" = %d\n", ret);
 
     if (ret < 0) {
+	port_dealloc(port_info);
 	errno = -ret;
 	return -1;
     }
-    files[fd].evtchn.ports[i].bound = 1;
-    files[fd].evtchn.ports[i].port = local_port;
+    port_info->bound = 1;
+    port_info->port = local_port;
     unmask_evtchn(local_port);
     return local_port;
 }
@@ -343,42 +352,40 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
 static int minios_evtchn_unbind(xc_evtchn *xce, xc_osdep_handle h, evtchn_port_t port)
 {
     int fd = (int)h;
-    int i;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++)
-	if (files[fd].evtchn.ports[i].port == port) {
-	    files[fd].evtchn.ports[i].port = -1;
-	    break;
-	}
-    if (i == MAX_EVTCHN_PORTS) {
-	printf("Warning: couldn't find port %"PRId32" for xc handle %x\n", port, fd);
-	errno = -EINVAL;
-	return -1;
+    struct evtchn_port_info *port_info;
+
+    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
+        if (port_info->port == port) {
+            port_dealloc(port_info);
+            return 0;
+        }
     }
-    files[fd].evtchn.ports[i].bound = 0;
-    unbind_evtchn(port);
-    return 0;
+    printf("Warning: couldn't find port %"PRId32" for xc handle %x\n", port, fd);
+    errno = -EINVAL;
+    return -1;
 }
 
 static evtchn_port_or_error_t minios_evtchn_bind_virq(xc_evtchn *xce, xc_osdep_handle h, unsigned int virq)
 {
     int fd = (int)h;
+    struct evtchn_port_info *port_info;
     evtchn_port_t port;
-    int i;
 
     assert(get_current() == main_thread);
-    i = port_alloc(fd);
-    if (i == -1)
+    port_info = port_alloc(fd);
+    if (port_info == NULL)
 	return -1;
 
     printf("xc_evtchn_bind_virq(%d)", virq);
     port = bind_virq(virq, evtchn_handler, (void*)(intptr_t)fd);
 
     if (port < 0) {
+	port_dealloc(port_info);
 	errno = -port;
 	return -1;
     }
-    files[fd].evtchn.ports[i].bound = 1;
-    files[fd].evtchn.ports[i].port = port;
+    port_info->bound = 1;
+    port_info->port = port;
     unmask_evtchn(port);
     return port;
 }
@@ -386,18 +393,18 @@ static evtchn_port_or_error_t minios_evtchn_bind_virq(xc_evtchn *xce, xc_osdep_h
 static evtchn_port_or_error_t minios_evtchn_pending(xc_evtchn *xce, xc_osdep_handle h)
 {
     int fd = (int)h;
-    int i;
+    struct evtchn_port_info *port_info;
     unsigned long flags;
     evtchn_port_t ret = -1;
 
     local_irq_save(flags);
     files[fd].read = 0;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++) {
-        evtchn_port_t port = files[fd].evtchn.ports[i].port;
-        if (port != -1 && files[fd].evtchn.ports[i].pending) {
+
+    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
+        if (port_info->port != -1 && port_info->pending) {
             if (ret == -1) {
-                ret = port;
-                files[fd].evtchn.ports[i].pending = 0;
+                ret = port_info->port;
+                port_info->pending = 0;
             } else {
                 files[fd].read = 1;
                 break;
-- 
1.7.7.5

[PATCH 10/18] xenstored: use grant references instead of map_foreign_range

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

make xenstored use grantref rather than map_foreign_range (which can
only be used by privileged domains)

This patch modifies the xenstore daemon to use xc_gnttab_map_grant_ref
instead of xc_map_foreign_range where available.

Previous versions of this patch have been sent to xen-devel. See
http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00610.html
http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01492.html

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_domain.c |   45 ++++++++++++++++++++++++++++++++-----
 1 files changed, 39 insertions(+), 6 deletions(-)

diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 443af82..0b8353b 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -32,8 +32,10 @@
 #include "xenstored_watch.h"
 
 #include <xenctrl.h>
+#include <xen/grant_table.h>
 
 static xc_interface **xc_handle;
+static xc_gnttab **xcg_handle;
 static evtchn_port_t virq_port;
 
 xc_evtchn *xce_handle = NULL;
@@ -174,8 +176,12 @@ static int destroy_domain(void *_domain)
 			eprintf("> Unbinding port %i failed!\n", domain->port);
 	}
 
-	if (domain->interface)
-		munmap(domain->interface, getpagesize());
+	if (domain->interface) {
+		if (*xcg_handle >= 0 && domain->domid != 0)
+			xc_gnttab_munmap(*xcg_handle, domain->interface, 1);
+		else
+			munmap(domain->interface, getpagesize());
+	}
 
 	fire_watches(NULL, "@releaseDomain", false);
 
@@ -344,9 +350,17 @@ void do_introduce(struct connection *conn, struct buffered_data *in)
 	domain = find_domain_by_domid(domid);
 
 	if (domain == NULL) {
-		interface = xc_map_foreign_range(
-			*xc_handle, domid,
-			getpagesize(), PROT_READ|PROT_WRITE, mfn);
+		if (*xcg_handle >= 0) {
+			/* this is the preferred method */
+			interface = xc_gnttab_map_grant_ref(
+				*xcg_handle, domid,
+				GNTTAB_RESERVED_XENSTORE,
+				PROT_READ|PROT_WRITE);
+		} else {
+			interface = xc_map_foreign_range(
+				*xc_handle, domid,
+				getpagesize(), PROT_READ|PROT_WRITE, mfn);
+		}
 		if (!interface) {
 			send_error(conn, errno);
 			return;
@@ -354,7 +368,10 @@ void do_introduce(struct connection *conn, struct buffered_data *in)
 		/* Hang domain off "in" until we're finished. */
 		domain = new_domain(in, domid, port);
 		if (!domain) {
-			munmap(interface, getpagesize());
+			if (*xcg_handle >= 0)
+				xc_gnttab_munmap(*xcg_handle, interface, 1);
+			else
+				munmap(interface, getpagesize());
 			send_error(conn, errno);
 			return;
 		}
@@ -552,6 +569,12 @@ static int close_xc_handle(void *_handle)
 	return 0;
 }
 
+static int close_xcg_handle(void *_handle)
+{
+	xc_gnttab_close(*(xc_gnttab **)_handle);
+	return 0;
+}
+
 /* Returns the implicit path of a connection (only domains have this) */
 const char *get_implicit_path(const struct connection *conn)
 {
@@ -603,6 +626,16 @@ void domain_init(void)
 
 	talloc_set_destructor(xc_handle, close_xc_handle);
 
+	xcg_handle = talloc(talloc_autofree_context(), xc_gnttab*);
+	if (!xcg_handle)
+		barf_perror("Failed to allocate domain gnttab handle");
+
+	*xcg_handle = xc_gnttab_open(NULL, 0);
+	if (*xcg_handle < 0)
+		xprintf("WARNING: Failed to open connection to gnttab\n");
+	else
+		talloc_set_destructor(xcg_handle, close_xcg_handle);
+
 	xce_handle = xc_evtchn_open(NULL, 0);
 
 	if (xce_handle == NULL)
-- 
1.7.7.5

[PATCH 11/18] xenstored: add NO_SOCKETS compilation option

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

option for compiling xenstored without unix sockets to support running on mini-OS

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c |   35 +++++++++++++++++++++++++++++++----
 tools/xenstore/xs.c             |    2 ++
 tools/xenstore/xs_lib.c         |    4 ++++
 3 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 9e6c2c7..0623aac 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -19,9 +19,11 @@
 
 #include <sys/types.h>
 #include <sys/stat.h>
-#include <sys/socket.h>
 #include <sys/select.h>
+#ifndef NO_SOCKETS
+#include <sys/socket.h>
 #include <sys/un.h>
+#endif
 #include <sys/time.h>
 #include <time.h>
 #include <unistd.h>
@@ -308,7 +310,10 @@ static void set_fd(int fd, fd_set *set, int *max)
 }
 
 
-static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
+static int initialize_set(fd_set *inset, fd_set *outset,
+#ifndef NO_SOCKETS
+			  int sock, int ro_sock,
+#endif
 			  struct timeval **ptimeout)
 {
 	static struct timeval zero_timeout = { 0 };
@@ -320,8 +325,10 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
 	FD_ZERO(inset);
 	FD_ZERO(outset);
 
+#ifndef NO_SOCKETS
 	set_fd(sock,               inset, &max);
 	set_fd(ro_sock,            inset, &max);
+#endif
 	set_fd(reopen_log_pipe[0], inset, &max);
 
 	if (xce_handle != NULL)
@@ -343,12 +350,14 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
 	return max;
 }
 
+#ifndef NO_SOCKETS
 static int destroy_fd(void *_fd)
 {
 	int *fd = _fd;
 	close(*fd);
 	return 0;
 }
+#endif
 
 /* Is child a subnode of parent, or equal? */
 bool is_child(const char *child, const char *parent)
@@ -1352,6 +1361,7 @@ struct connection *new_connection(connwritefn_t *write, connreadfn_t *read)
 	return new;
 }
 
+#ifndef NO_SOCKETS
 static int writefd(struct connection *conn, const void *data, unsigned int len)
 {
 	int rc;
@@ -1406,6 +1416,7 @@ static void accept_connection(int sock, bool canwrite)
 	} else
 		close(fd);
 }
+#endif
 
 #define TDB_FLAGS 0
 
@@ -1752,8 +1763,11 @@ extern void dump_conn(struct connection *conn);
 
 int main(int argc, char *argv[])
 {
-	int opt, *sock, *ro_sock, max;
+	int opt, max;
+#ifndef NO_SOCKETS
+	int *sock, *ro_sock;
 	struct sockaddr_un addr;
+#endif
 	fd_set inset, outset;
 	bool dofork = true;
 	bool outputpid = false;
@@ -1837,6 +1851,7 @@ int main(int argc, char *argv[])
 	if (!dofork)
 		talloc_enable_leak_report_full();
 
+#ifndef NO_SOCKETS
 	/* Create sockets for them to listen to. */
 	sock = talloc(talloc_autofree_context(), int);
 	*sock = socket(PF_UNIX, SOCK_STREAM, 0);
@@ -1848,10 +1863,12 @@ int main(int argc, char *argv[])
 		barf_perror("Could not create socket");
 	talloc_set_destructor(sock, destroy_fd);
 	talloc_set_destructor(ro_sock, destroy_fd);
+#endif
 
 	/* Don't kill us with SIGPIPE. */
 	signal(SIGPIPE, SIG_IGN);
 
+#ifndef NO_SOCKETS
 	/* FIXME: Be more sophisticated, don't mug running daemon. */
 	unlink(xs_daemon_socket());
 	unlink(xs_daemon_socket_ro());
@@ -1871,6 +1888,7 @@ int main(int argc, char *argv[])
 	if (listen(*sock, 1) != 0
 	    || listen(*ro_sock, 1) != 0)
 		barf_perror("Could not listen on sockets");
+#endif
 
 	if (pipe(reopen_log_pipe)) {
 		barf_perror("pipe");
@@ -1909,7 +1927,11 @@ int main(int argc, char *argv[])
 		evtchn_fd = xc_evtchn_fd(xce_handle);
 
 	/* Get ready to listen to the tools. */
+#ifndef NO_SOCKETS
 	max = initialize_set(&inset, &outset, *sock, *ro_sock, &timeout);
+#else
+	max = initialize_set(&inset, &outset, &timeout);
+#endif
 
 	/* Tell the kernel we're up and running. */
 	xenbus_notify_running();
@@ -1931,11 +1953,13 @@ int main(int argc, char *argv[])
 			reopen_log();
 		}
 
+#ifndef NO_SOCKETS
 		if (FD_ISSET(*sock, &inset))
 			accept_connection(*sock, true);
 
 		if (FD_ISSET(*ro_sock, &inset))
 			accept_connection(*ro_sock, false);
+#endif
 
 		if (evtchn_fd != -1 && FD_ISSET(evtchn_fd, &inset))
 			handle_event();
@@ -1977,7 +2001,10 @@ int main(int argc, char *argv[])
 			}
 		}
 
-		max = initialize_set(&inset, &outset, *sock, *ro_sock,
+		max = initialize_set(&inset, &outset,
+#ifndef NO_SOCKETS
+				     *sock, *ro_sock,
+#endif
 				     &timeout);
 	}
 }
diff --git a/tools/xenstore/xs.c b/tools/xenstore/xs.c
index 8e54fe0..119d945 100644
--- a/tools/xenstore/xs.c
+++ b/tools/xenstore/xs.c
@@ -271,10 +271,12 @@ struct xs_handle *xs_open(unsigned long flags)
 {
 	struct xs_handle *xsh = NULL;
 
+#ifndef NO_SOCKETS
 	if (flags & XS_OPEN_READONLY)
 		xsh = get_handle(xs_daemon_socket_ro());
 	else
 		xsh = get_handle(xs_daemon_socket());
+#endif
 
 	if (!xsh && !(flags & XS_OPEN_SOCKETONLY))
 		xsh = get_handle(xs_domain_dev());
diff --git a/tools/xenstore/xs_lib.c b/tools/xenstore/xs_lib.c
index 03a9ee4..af3db6b 100644
--- a/tools/xenstore/xs_lib.c
+++ b/tools/xenstore/xs_lib.c
@@ -39,6 +39,7 @@ const char *xs_daemon_rundir(void)
 	return (s ? s : "/var/run/xenstored");
 }
 
+#ifndef NO_SOCKETS
 static const char *xs_daemon_path(void)
 {
 	static char buf[PATH_MAX];
@@ -50,6 +51,7 @@ static const char *xs_daemon_path(void)
 		return NULL;
 	return buf;
 }
+#endif
 
 const char *xs_daemon_tdb(void)
 {
@@ -58,6 +60,7 @@ const char *xs_daemon_tdb(void)
 	return buf;
 }
 
+#ifndef NO_SOCKETS
 const char *xs_daemon_socket(void)
 {
 	return xs_daemon_path();
@@ -73,6 +76,7 @@ const char *xs_daemon_socket_ro(void)
 		return NULL;
 	return buf;
 }
+#endif
 
 const char *xs_domain_dev(void)
 {
-- 
1.7.7.5

Re: [PATCH 11/18] xenstored: add NO_SOCKETS compilation option

[Ian Campbell]

On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> @@ -308,7 +310,10 @@ static void set_fd(int fd, fd_set *set, int *max)
>  }
>  
>  
> -static int initialize_set(fd_set *inset, fd_set *outset, int sock,
> int ro_sock,
> +static int initialize_set(fd_set *inset, fd_set *outset,
> +#ifndef NO_SOCKETS
> +                         int sock, int ro_sock,
> +#endif
>                           struct timeval **ptimeout)
>  {
>         static struct timeval zero_timeout = { 0 }; 

Rather than removing the argument why not simply accept -1 as a "no
sock" value and DTWT with that, this ought to reduce the amount of
ifdeffery substantially

Ian.

[PATCH 12/18] xenstored support for in-memory rather than FS based trivial DB (needed to run on mini-OS)

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

tdb_copy (a xen modification to tdb?) should honor the TDB_INTERNAL flag
for in-memory databases.

TODO: leaks memory on error case

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/tdb.c |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/tools/xenstore/tdb.c b/tools/xenstore/tdb.c
index 63205e1..639ce6e 100644
--- a/tools/xenstore/tdb.c
+++ b/tools/xenstore/tdb.c
@@ -2103,6 +2103,42 @@ TDB_CONTEXT *tdb_copy(TDB_CONTEXT *tdb, const char *outfile)
 	int fd, saved_errno;
 	TDB_CONTEXT *copy;
 
+	if (tdb->flags & TDB_INTERNAL) {
+		struct tdb_header *copydb;
+		
+		copy = talloc_zero(outfile, TDB_CONTEXT);
+		if (copy == NULL) {
+			errno = ENOMEM;
+			goto intfail;
+		}
+		memcpy(copy, tdb, sizeof(TDB_CONTEXT));
+
+		if (copy->name || copy->locked || copy->device || copy->inode) {
+			fprintf(stderr, "tdb_copy assumption(s) failed\n");
+			goto intfail;
+		}
+
+		copydb = talloc_zero_size(copy, copy->map_size);
+		if (copydb == NULL) {
+			errno = ENOMEM;
+			goto intfail;
+		}
+		memcpy(copydb, copy->map_ptr, copy->map_size);
+		copy->map_ptr = (char*) copydb;
+
+		if (tdb_brlock(tdb, GLOBAL_LOCK, F_UNLCK, F_SETLKW, 0) == -1)
+			goto intfail;
+
+		copy->next = tdbs;
+		tdbs = copy;
+
+
+		return copy;
+intfail:
+		/* TODO (leaking memory is easier) */
+		return NULL;
+	}
+
 	fd = open(outfile, O_TRUNC|O_CREAT|O_WRONLY, 0640);
 	if (fd < 0)
 		return NULL;
-- 
1.7.7.5

[PATCH 13/18] xenstored: support running in minios stubdom

[Daniel De Graaf]

A previous versions of this patch has been sent to xen-devel. See
http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01655.html

Originally-by: Diego Ongaro <diego.ongaro@citrix.com>
Originally-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/include/list.h          |    6 ++--
 extras/mini-os/main.c                  |    4 +++
 stubdom/Makefile                       |   29 +++++++++++++++++--
 tools/xenstore/Makefile                |    9 +++++-
 tools/xenstore/tdb.c                   |    6 ++--
 tools/xenstore/utils.h                 |    2 +
 tools/xenstore/xenstored_core.c        |   47 ++++++++++++++++++++++++++++++-
 tools/xenstore/xenstored_domain.c      |    7 +++++
 tools/xenstore/xenstored_transaction.c |    2 +
 9 files changed, 100 insertions(+), 12 deletions(-)

diff --git a/extras/mini-os/include/list.h b/extras/mini-os/include/list.h
index a60ae23..4e6a2ac 100644
--- a/extras/mini-os/include/list.h
+++ b/extras/mini-os/include/list.h
@@ -1,5 +1,5 @@
-#ifndef _LINUX_LIST_H
-#define _LINUX_LIST_H
+#ifndef _MINIOS_LIST_H
+#define _MINIOS_LIST_H
 
 /*
  * Simple doubly linked list implementation.
@@ -186,5 +186,5 @@ static __inline__ void minios_list_splice(struct minios_list_head *list, struct
 		n = minios_list_entry(pos->member.next, typeof(*pos), member);	\
 	     &pos->member != (head); 					\
 	     pos = n, n = minios_list_entry(n->member.next, typeof(*n), member))
-#endif /* _LINUX_LIST_H */
+#endif /* _MINIOS_LIST_H */
 
diff --git a/extras/mini-os/main.c b/extras/mini-os/main.c
index b95b889..cd89849 100644
--- a/extras/mini-os/main.c
+++ b/extras/mini-os/main.c
@@ -60,6 +60,7 @@ static void call_main(void *p)
      * crashing. */
     //sleep(1);
 
+#ifndef CONFIG_XENSTORE
 #ifndef CONFIG_GRUB
     sparse((unsigned long) &__app_bss_start, &__app_bss_end - &__app_bss_start);
 #if defined(HAVE_LWIP) && !defined(CONFIG_QEMU)
@@ -67,6 +68,7 @@ static void call_main(void *p)
 #endif
 #endif
     create_thread("pcifront", pcifront_watches, NULL);
+#endif
 
 #ifdef CONFIG_QEMU
     /* Fetch argc, argv from XenStore */
@@ -169,9 +171,11 @@ void _exit(int ret)
     close_all_files();
     __libc_fini_array();
     printk("main returned %d\n", ret);
+#ifndef CONFIG_XENSTORE
 #ifdef HAVE_LWIP
     stop_networking();
 #endif
+#endif
     stop_kernel();
     if (!ret) {
 	/* No problem, just shutdown.  */
diff --git a/stubdom/Makefile b/stubdom/Makefile
index 3705059..e0a90a9 100644
--- a/stubdom/Makefile
+++ b/stubdom/Makefile
@@ -74,14 +74,14 @@ TARGET_CPPFLAGS += -I$(XEN_ROOT)/xen/include
 
 TARGET_LDFLAGS += -nostdlib -L$(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/lib
 
-TARGETS=ioemu c caml grub
+TARGETS=ioemu c caml grub xenstore
 
 CROSS_MAKE := $(MAKE) DESTDIR=
 
 .PHONY: all
 all: build
 ifeq ($(STUBDOM_SUPPORTED),1)
-build: genpath ioemu-stubdom c-stubdom pv-grub
+build: genpath ioemu-stubdom c-stubdom pv-grub xenstore-stubdom
 else
 build: genpath
 endif
@@ -262,6 +262,11 @@ mk-headers-$(XEN_TARGET_ARCH): ioemu/linkfarm.stamp
 	  ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.c . && \
 	  ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.h . && \
 	  ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/Makefile . )
+	mkdir -p xenstore
+	[ -h xenstore/Makefile ] || ( cd xenstore && \
+	  ln -sf $(XEN_ROOT)/tools/xenstore/*.c . && \
+	  ln -sf $(XEN_ROOT)/tools/xenstore/*.h . && \
+	  ln -sf $(XEN_ROOT)/tools/xenstore/Makefile . )
 	$(CROSS_MAKE) -C $(MINI_OS) links
 	touch mk-headers-$(XEN_TARGET_ARCH)
 
@@ -334,6 +339,14 @@ grub: grub-upstream $(CROSS_ROOT)
 	mkdir -p grub-$(XEN_TARGET_ARCH)
 	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(CROSS_MAKE) -C $@ OBJ_DIR=$(CURDIR)/grub-$(XEN_TARGET_ARCH)
 
+##########
+# xenstore
+##########
+
+.PHONY: xenstore
+xenstore: $(CROSS_ROOT)
+	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(CROSS_MAKE) -C $@ LWIPDIR=$(CURDIR)/lwip xenstored.a CONFIG_STUBDOM=y
+
 ########
 # minios
 ########
@@ -355,12 +368,16 @@ c-stubdom: mini-os-$(XEN_TARGET_ARCH)-c lwip-$(XEN_TARGET_ARCH) libxc c
 pv-grub: mini-os-$(XEN_TARGET_ARCH)-grub libxc grub
 	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(CROSS_MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
 
+.PHONY: xenstore-stubdom
+xenstore-stubdom: mini-os-$(XEN_TARGET_ARCH)-xenstore libxc xenstore
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_XENSTORE $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) APP_OBJS=$(CURDIR)/xenstore/xenstored.a
+
 #########
 # install
 #########
 
 ifeq ($(STUBDOM_SUPPORTED),1)
-install: genpath install-readme install-ioemu install-grub
+install: genpath install-readme install-ioemu install-grub install-xenstore
 else
 install: genpath
 endif
@@ -379,6 +396,10 @@ install-grub: pv-grub
 	$(INSTALL_DIR) "$(DESTDIR)$(XENFIRMWAREDIR)"
 	$(INSTALL_DATA) mini-os-$(XEN_TARGET_ARCH)-grub/mini-os.gz "$(DESTDIR)$(XENFIRMWAREDIR)/pv-grub-$(XEN_TARGET_ARCH).gz"
 
+install-xenstore: xenstore-stubdom
+	$(INSTALL_DIR) "$(DESTDIR)/usr/lib/xen/boot"
+	$(INSTALL_PROG) mini-os-$(XEN_TARGET_ARCH)-xenstore/mini-os.gz "$(DESTDIR)/usr/lib/xen/boot/xenstore-stubdom.gz"
+
 #######
 # clean
 #######
@@ -390,12 +411,14 @@ clean:
 	rm -fr mini-os-$(XEN_TARGET_ARCH)-c
 	rm -fr mini-os-$(XEN_TARGET_ARCH)-caml
 	rm -fr mini-os-$(XEN_TARGET_ARCH)-grub
+	rm -fr mini-os-$(XEN_TARGET_ARCH)-xenstore
 	$(CROSS_MAKE) -C caml clean
 	$(CROSS_MAKE) -C c clean
 	rm -fr grub-$(XEN_TARGET_ARCH)
 	rm -f $(STUBDOMPATH)
 	[ ! -d libxc-$(XEN_TARGET_ARCH) ] || $(CROSS_MAKE) -C libxc-$(XEN_TARGET_ARCH) clean
 	-[ ! -d ioemu ] || $(CROSS_MAKE) -C ioemu clean
+	-[ ! -d xenstore ] || $(CROSS_MAKE) -C xenstore clean
 
 # clean the cross-compilation result
 .PHONY: crossclean
diff --git a/tools/xenstore/Makefile b/tools/xenstore/Makefile
index 4facb62..3a061d6 100644
--- a/tools/xenstore/Makefile
+++ b/tools/xenstore/Makefile
@@ -28,6 +28,10 @@ endif
 
 ALL_TARGETS = libxenstore.so libxenstore.a clients xs_tdb_dump xenstored
 
+ifdef CONFIG_STUBDOM
+CFLAGS += -DNO_SOCKETS=1 -DNO_LOCAL_XENBUS=1 -DNO_SYSLOG=1 -DNO_REOPEN_LOG=1
+endif
+
 .PHONY: all
 all: $(ALL_TARGETS)
 
@@ -45,10 +49,13 @@ xenstored_probes.o: xenstored_solaris.o
 
 CFLAGS += -DHAVE_DTRACE=1
 endif
- 
+
 xenstored: $(XENSTORED_OBJS)
 	$(CC) $(LDFLAGS) $^ $(LDLIBS_libxenctrl) $(SOCKET_LIBS) -o $@ $(APPEND_LDFLAGS)
 
+xenstored.a: $(XENSTORED_OBJS)
+	$(AR) cr $@ $^
+
 $(CLIENTS): xenstore
 	ln -f xenstore $@
 
diff --git a/tools/xenstore/tdb.c b/tools/xenstore/tdb.c
index 639ce6e..75ffd2a 100644
--- a/tools/xenstore/tdb.c
+++ b/tools/xenstore/tdb.c
@@ -1334,7 +1334,7 @@ static int tdb_next_lock(TDB_CONTEXT *tdb, struct tdb_traverse_lock *tlock,
 
 		/* Iterate through chain */
 		while( tlock->off) {
-			tdb_off current;
+			tdb_off mycurrent;
 			if (rec_read(tdb, tlock->off, rec) == -1)
 				goto fail;
 
@@ -1352,10 +1352,10 @@ static int tdb_next_lock(TDB_CONTEXT *tdb, struct tdb_traverse_lock *tlock,
 			}
 
 			/* Try to clean dead ones from old traverses */
-			current = tlock->off;
+			mycurrent = tlock->off;
 			tlock->off = rec->next;
 			if (!tdb->read_only && 
-			    do_delete(tdb, current, rec) != 0)
+			    do_delete(tdb, mycurrent, rec) != 0)
 				goto fail;
 		}
 		tdb_unlock(tdb, tlock->hash, F_WRLCK);
diff --git a/tools/xenstore/utils.h b/tools/xenstore/utils.h
index f378343..2effd17 100644
--- a/tools/xenstore/utils.h
+++ b/tools/xenstore/utils.h
@@ -19,7 +19,9 @@ static inline bool strends(const char *a, const char *b)
 	return streq(a + strlen(a) - strlen(b), b);
 }
 
+#ifndef ARRAY_SIZE
 #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
+#endif
 
 void barf(const char *fmt, ...) __attribute__((noreturn));
 void barf_perror(const char *fmt, ...) __attribute__((noreturn));
diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 0623aac..2d4a7e1 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -32,7 +32,9 @@
 #include <stdio.h>
 #include <stdarg.h>
 #include <stdlib.h>
+#ifndef NO_SYSLOG
 #include <syslog.h>
+#endif
 #include <string.h>
 #include <errno.h>
 #include <dirent.h>
@@ -61,13 +63,24 @@ LIST_HEAD(connections);
 static int tracefd = -1;
 static bool recovery = true;
 static bool remove_local = true;
+#ifndef NO_REOPEN_LOG
 static int reopen_log_pipe[2];
+#endif
 static char *tracefile = NULL;
 static TDB_CONTEXT *tdb_ctx;
 
 static void corrupt(struct connection *conn, const char *fmt, ...);
 static void check_store(void);
 
+#ifdef __MINIOS__
+#define lockf(...) (-ENOSYS)
+#endif
+
+#ifdef NO_SYSLOG
+#define openlog(...) ((void) 0)
+#define syslog(...)  ((void) 0)
+#endif
+
 #define log(...)							\
 	do {								\
 		char *s = talloc_asprintf(NULL, __VA_ARGS__);		\
@@ -92,8 +105,10 @@ TDB_CONTEXT *tdb_context(struct connection *conn)
 
 bool replace_tdb(const char *newname, TDB_CONTEXT *newtdb)
 {
+#ifndef __MINIOS__
 	if (rename(newname, xs_daemon_tdb()) != 0)
 		return false;
+#endif
 	tdb_close(tdb_ctx);
 	tdb_ctx = talloc_steal(talloc_autofree_context(), newtdb);
 	return true;
@@ -195,6 +210,7 @@ void trace_destroy(const void *data, const char *type)
 	trace("DESTROY %s %p\n", type, data);
 }
 
+#ifndef NO_REOPEN_LOG
 /**
  * Signal handler for SIGHUP, which requests that the trace log is reopened
  * (in the main loop).  A single byte is written to reopen_log_pipe, to awaken
@@ -222,7 +238,7 @@ static void reopen_log(void)
 			trace("\n***\n");
 	}
 }
-
+#endif
 
 static bool write_messages(struct connection *conn)
 {
@@ -329,7 +345,9 @@ static int initialize_set(fd_set *inset, fd_set *outset,
 	set_fd(sock,               inset, &max);
 	set_fd(ro_sock,            inset, &max);
 #endif
+#ifndef NO_REOPEN_LOG
 	set_fd(reopen_log_pipe[0], inset, &max);
+#endif
 
 	if (xce_handle != NULL)
 		set_fd(xc_evtchn_fd(xce_handle), inset, &max);
@@ -1418,7 +1436,11 @@ static void accept_connection(int sock, bool canwrite)
 }
 #endif
 
+#ifdef __MINIOS__
+#define TDB_FLAGS TDB_INTERNAL|TDB_NOLOCK
+#else
 #define TDB_FLAGS 0
+#endif
 
 /* We create initial nodes manually. */
 static void manual_node(const char *name, const char *child)
@@ -1443,7 +1465,11 @@ static void setup_structure(void)
 {
 	char *tdbname;
 	tdbname = talloc_strdup(talloc_autofree_context(), xs_daemon_tdb());
+#ifdef __MINIOS__
+	tdb_ctx = NULL;
+#else
 	tdb_ctx = tdb_open(tdbname, 0, TDB_FLAGS, O_RDWR, 0);
+#endif
 
 	if (tdb_ctx) {
 		/* XXX When we make xenstored able to restart, this will have
@@ -1669,6 +1695,7 @@ static void corrupt(struct connection *conn, const char *fmt, ...)
 }
 
 
+#ifndef __MINIOS__
 static void write_pidfile(const char *pidfile)
 {
 	char buf[100];
@@ -1715,7 +1742,7 @@ static void daemonize(void)
 	/* Discard our parent's old-fashioned umask prejudices. */
 	umask(0);
 }
-
+#endif
 
 static void usage(void)
 {
@@ -1769,7 +1796,11 @@ int main(int argc, char *argv[])
 	struct sockaddr_un addr;
 #endif
 	fd_set inset, outset;
+#ifdef __MINIOS__
+	bool dofork = false;
+#else
 	bool dofork = true;
+#endif
 	bool outputpid = false;
 	bool no_domain_init = false;
 	const char *pidfile = NULL;
@@ -1823,8 +1854,11 @@ int main(int argc, char *argv[])
 	if (optind != argc)
 		barf("%s: No arguments desired", argv[0]);
 
+#ifndef NO_REOPEN_LOG
 	reopen_log();
+#endif
 
+#ifndef __MINIOS__
 	/* make sure xenstored directory exists */
 	if (mkdir(xs_daemon_rundir(), 0755)) {
 		if (errno != EEXIST) {
@@ -1846,6 +1880,7 @@ int main(int argc, char *argv[])
 	}
 	if (pidfile)
 		write_pidfile(pidfile);
+#endif
 
 	/* Talloc leak reports go to stderr, which is closed if we fork. */
 	if (!dofork)
@@ -1890,9 +1925,11 @@ int main(int argc, char *argv[])
 		barf_perror("Could not listen on sockets");
 #endif
 
+#ifndef NO_REOPEN_LOG
 	if (pipe(reopen_log_pipe)) {
 		barf_perror("pipe");
 	}
+#endif
 
 	/* Setup the database */
 	setup_structure();
@@ -1921,7 +1958,9 @@ int main(int argc, char *argv[])
 		xprintf = trace;
 	}
 
+#ifndef NO_REOPEN_LOG
 	signal(SIGHUP, trigger_reopen_log);
+#endif
 
 	if (xce_handle != NULL)
 		evtchn_fd = xc_evtchn_fd(xce_handle);
@@ -1933,8 +1972,10 @@ int main(int argc, char *argv[])
 	max = initialize_set(&inset, &outset, &timeout);
 #endif
 
+#ifndef __MINIOS__
 	/* Tell the kernel we're up and running. */
 	xenbus_notify_running();
+#endif
 
 	/* Main loop. */
 	for (;;) {
@@ -1946,12 +1987,14 @@ int main(int argc, char *argv[])
 			barf_perror("Select failed");
 		}
 
+#ifndef NO_REOPEN_LOG
 		if (FD_ISSET(reopen_log_pipe[0], &inset)) {
 			char c;
 			if (read(reopen_log_pipe[0], &c, 1) != 1)
 				barf_perror("read failed");
 			reopen_log();
 		}
+#endif
 
 #ifndef NO_SOCKETS
 		if (FD_ISSET(*sock, &inset))
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 0b8353b..811ae3c 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -588,6 +588,12 @@ void restore_existing_connections(void)
 {
 }
 
+#ifdef __MINIOS__
+static inline int dom0_init(void) 
+{
+	return 0;
+}
+#else
 static int dom0_init(void) 
 { 
 	evtchn_port_t port;
@@ -611,6 +617,7 @@ static int dom0_init(void)
 
 	return 0; 
 }
+#endif
 
 void domain_init(void)
 {
diff --git a/tools/xenstore/xenstored_transaction.c b/tools/xenstore/xenstored_transaction.c
index 380c691..c59acfb 100644
--- a/tools/xenstore/xenstored_transaction.c
+++ b/tools/xenstore/xenstored_transaction.c
@@ -120,7 +120,9 @@ static int destroy_transaction(void *_transaction)
 	trace_destroy(trans, "transaction");
 	if (trans->tdb)
 		tdb_close(trans->tdb);
+#ifndef __MINIOS__
 	unlink(trans->tdb_name);
+#endif
 	return 0;
 }
 
-- 
1.7.7.5

[PATCH 14/18] xenstored: always use xc_gnttab_munmap in stubdom

[Daniel De Graaf]

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_domain.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 811ae3c..aca2149 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -177,10 +177,14 @@ static int destroy_domain(void *_domain)
 	}
 
 	if (domain->interface) {
+#ifdef __MINIOS__
+		xc_gnttab_munmap(*xcg_handle, domain->interface, 1);
+#else
 		if (*xcg_handle >= 0 && domain->domid != 0)
 			xc_gnttab_munmap(*xcg_handle, domain->interface, 1);
 		else
 			munmap(domain->interface, getpagesize());
+#endif
 	}
 
 	fire_watches(NULL, "@releaseDomain", false);
-- 
1.7.7.5

[PATCH 15/18] xenstored: add --event parameter for bootstrapping

[Daniel De Graaf]

When xenstored is run in a minios domain, it needs a bootstrap
connection to dom0 so that additional domain introduce messages can be
sent to it.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c   |    5 +++++
 tools/xenstore/xenstored_core.h   |    1 +
 tools/xenstore/xenstored_domain.c |   13 ++++++++++++-
 3 files changed, 18 insertions(+), 1 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 2d4a7e1..7b63a68 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -1774,6 +1774,7 @@ static struct option options[] = {
 	{ "no-domain-init", 0, NULL, 'D' },
 	{ "entry-nb", 1, NULL, 'E' },
 	{ "pid-file", 1, NULL, 'F' },
+	{ "event", 1, NULL, 'e' },
 	{ "help", 0, NULL, 'H' },
 	{ "no-fork", 0, NULL, 'N' },
 	{ "output-pid", 0, NULL, 'P' },
@@ -1787,6 +1788,7 @@ static struct option options[] = {
 	{ NULL, 0, NULL, 0 } };
 
 extern void dump_conn(struct connection *conn); 
+int dom0_event = 0;
 
 int main(int argc, char *argv[])
 {
@@ -1849,6 +1851,9 @@ int main(int argc, char *argv[])
 		case 'W':
 			quota_nb_watch_per_domain = strtol(optarg, NULL, 10);
 			break;
+		case 'e':
+			dom0_event = strtol(optarg, NULL, 10);
+			break;
 		}
 	}
 	if (optind != argc)
diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
index c487089..d3040ba 100644
--- a/tools/xenstore/xenstored_core.h
+++ b/tools/xenstore/xenstored_core.h
@@ -168,6 +168,7 @@ void trace(const char *fmt, ...);
 void dtrace_io(const struct connection *conn, const struct buffered_data *data, int out);
 
 extern int event_fd;
+extern int dom0_event;
 
 /* Map the kernel's xenstore page. */
 void *xenbus_map(void);
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index aca2149..648eb1d 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -593,8 +593,19 @@ void restore_existing_connections(void)
 }
 
 #ifdef __MINIOS__
-static inline int dom0_init(void) 
+static int dom0_init(void)
 {
+	struct domain *domain;
+	int domid = 0;
+	evtchn_port_t port = dom0_event;
+
+	domain = new_domain(NULL, domid, port);
+	domain->interface = xc_gnttab_map_grant_ref(*xcg_handle, domid,
+			GNTTAB_RESERVED_XENSTORE, PROT_READ|PROT_WRITE);
+	talloc_steal(domain->conn, domain);
+
+	xc_evtchn_notify(xce_handle, domain->port);
+
 	return 0;
 }
 #else
-- 
1.7.7.5

[PATCH 16/18] xenstored: pull dom0 event port from shared page

[Daniel De Graaf]

When building the xenstored domain using xl, it is difficult to specify
the event channel number on the command line because the command line is
parsed prior to domain creation, while the event channel cannot be
created until after domain creation. To avoid needing a special domain
builder for the xenstore domain, store the event channel port in an
unused field of the shared page.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_domain.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 648eb1d..b100aa4 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -598,10 +598,18 @@ static int dom0_init(void)
 	struct domain *domain;
 	int domid = 0;
 	evtchn_port_t port = dom0_event;
+	void *interface;
 
-	domain = new_domain(NULL, domid, port);
-	domain->interface = xc_gnttab_map_grant_ref(*xcg_handle, domid,
+	interface = xc_gnttab_map_grant_ref(*xcg_handle, domid,
 			GNTTAB_RESERVED_XENSTORE, PROT_READ|PROT_WRITE);
+
+	if (!port) {
+		port = *(uint16_t*)(interface +
+			sizeof(struct xenstore_domain_interface));
+	}
+
+	domain = new_domain(NULL, domid, port);
+	domain->interface = interface;
 	talloc_steal(domain->conn, domain);
 
 	xc_evtchn_notify(xce_handle, domain->port);
-- 
1.7.7.5

[PATCH 17/18] xenstored: use domain_is_unprivileged instead of checking conn->id

[Daniel De Graaf]

This centralizes all the permission checking for privileged domains in
preparation for allowing domains other than dom0 to be privileged.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c   |    6 +++---
 tools/xenstore/xenstored_domain.c |    8 ++++----
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 7b63a68..65ceb2c 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -491,7 +491,7 @@ static enum xs_perm_type perm_for_conn(struct connection *conn,
 		mask &= ~XS_PERM_WRITE;
 
 	/* Owners and tools get it all... */
-	if (!conn->id || perms[0].id == conn->id
+	if (!domain_is_unprivileged(conn) || perms[0].id == conn->id
                 || (conn->target && perms[0].id == conn->target->id))
 		return (XS_PERM_READ|XS_PERM_WRITE|XS_PERM_OWNER) & mask;
 
@@ -829,11 +829,11 @@ static struct node *construct_node(struct connection *conn, const char *name)
 	node->tdb = tdb_context(conn);
 	node->name = talloc_strdup(node, name);
 
-	/* Inherit permissions, except domains own what they create */
+	/* Inherit permissions, except unprivileged domains own what they create */
 	node->num_perms = parent->num_perms;
 	node->perms = talloc_memdup(node, parent->perms,
 				    node->num_perms * sizeof(node->perms[0]));
-	if (conn && conn->id)
+	if (domain_is_unprivileged(conn))
 		node->perms[0].id = conn->id;
 
 	/* No children, no data */
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index b100aa4..5bf16e8 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -336,7 +336,7 @@ void do_introduce(struct connection *conn, struct buffered_data *in)
 		return;
 	}
 
-	if (conn->id != 0 || !conn->can_write) {
+	if (domain_is_unprivileged(conn) || !conn->can_write) {
 		send_error(conn, EACCES);
 		return;
 	}
@@ -413,7 +413,7 @@ void do_set_target(struct connection *conn, struct buffered_data *in)
 		return;
 	}
 
-	if (conn->id != 0 || !conn->can_write) {
+	if (domain_is_unprivileged(conn) || !conn->can_write) {
 		send_error(conn, EACCES);
 		return;
 	}
@@ -465,7 +465,7 @@ void do_release(struct connection *conn, const char *domid_str)
 		return;
 	}
 
-	if (conn->id != 0) {
+	if (domain_is_unprivileged(conn)) {
 		send_error(conn, EACCES);
 		return;
 	}
@@ -502,7 +502,7 @@ void do_resume(struct connection *conn, const char *domid_str)
 		return;
 	}
 
-	if (conn->id != 0) {
+	if (domain_is_unprivileged(conn)) {
 		send_error(conn, EACCES);
 		return;
 	}
-- 
1.7.7.5

[PATCH 18/18] xenstored: add --priv-domid parameter

[Daniel De Graaf]

This parameter identifies an alternative service domain which has
superuser access to the xenstore database, which is currently required
to set up a new domain's xenstore entries.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c   |    5 +++++
 tools/xenstore/xenstored_core.h   |    1 +
 tools/xenstore/xenstored_domain.c |    2 +-
 3 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 65ceb2c..4437c8d 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -1777,6 +1777,7 @@ static struct option options[] = {
 	{ "event", 1, NULL, 'e' },
 	{ "help", 0, NULL, 'H' },
 	{ "no-fork", 0, NULL, 'N' },
+	{ "priv-domid", 1, NULL, 'p' },
 	{ "output-pid", 0, NULL, 'P' },
 	{ "entry-size", 1, NULL, 'S' },
 	{ "trace-file", 1, NULL, 'T' },
@@ -1789,6 +1790,7 @@ static struct option options[] = {
 
 extern void dump_conn(struct connection *conn); 
 int dom0_event = 0;
+int priv_domid = 0;
 
 int main(int argc, char *argv[])
 {
@@ -1854,6 +1856,9 @@ int main(int argc, char *argv[])
 		case 'e':
 			dom0_event = strtol(optarg, NULL, 10);
 			break;
+		case 'p':
+			priv_domid = strtol(optarg, NULL, 10);
+			break;
 		}
 	}
 	if (optind != argc)
diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
index d3040ba..03e2e48 100644
--- a/tools/xenstore/xenstored_core.h
+++ b/tools/xenstore/xenstored_core.h
@@ -169,6 +169,7 @@ void dtrace_io(const struct connection *conn, const struct buffered_data *data,
 
 extern int event_fd;
 extern int dom0_event;
+extern int priv_domid;
 
 /* Map the kernel's xenstore page. */
 void *xenbus_map(void);
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 5bf16e8..ba9a5ef 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -241,7 +241,7 @@ bool domain_can_read(struct connection *conn)
 
 bool domain_is_unprivileged(struct connection *conn)
 {
-	return (conn && conn->domain && conn->domain->domid != 0);
+	return (conn && conn->domain && conn->domain->domid != 0 && conn->domain->domid != priv_domid);
 }
 
 bool domain_can_write(struct connection *conn)
-- 
1.7.7.5

Re: [PATCH 18/18] xenstored: add --priv-domid parameter

[Ian Campbell]

On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> This parameter identifies an alternative service domain which has
> superuser access to the xenstore database, which is currently required
> to set up a new domain's xenstore entries.
> 
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  tools/xenstore/xenstored_core.c   |    5 +++++
>  tools/xenstore/xenstored_core.h   |    1 +
>  tools/xenstore/xenstored_domain.c |    2 +-
>  3 files changed, 7 insertions(+), 1 deletions(-)
> 
> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
> index 65ceb2c..4437c8d 100644
> --- a/tools/xenstore/xenstored_core.c
> +++ b/tools/xenstore/xenstored_core.c
> @@ -1777,6 +1777,7 @@ static struct option options[] = {
>  	{ "event", 1, NULL, 'e' },
>  	{ "help", 0, NULL, 'H' },
>  	{ "no-fork", 0, NULL, 'N' },
> +	{ "priv-domid", 1, NULL, 'p' },
>  	{ "output-pid", 0, NULL, 'P' },
>  	{ "entry-size", 1, NULL, 'S' },
>  	{ "trace-file", 1, NULL, 'T' },
> @@ -1789,6 +1790,7 @@ static struct option options[] = {
>  
>  extern void dump_conn(struct connection *conn); 
>  int dom0_event = 0;
> +int priv_domid = 0;
>  
>  int main(int argc, char *argv[])
>  {
> @@ -1854,6 +1856,9 @@ int main(int argc, char *argv[])
>  		case 'e':
>  			dom0_event = strtol(optarg, NULL, 10);
>  			break;
> +		case 'p':
> +			priv_domid = strtol(optarg, NULL, 10);
> +			break;
>  		}
>  	}
>  	if (optind != argc)
> diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
> index d3040ba..03e2e48 100644
> --- a/tools/xenstore/xenstored_core.h
> +++ b/tools/xenstore/xenstored_core.h
> @@ -169,6 +169,7 @@ void dtrace_io(const struct connection *conn, const struct buffered_data *data,
>  
>  extern int event_fd;
>  extern int dom0_event;
> +extern int priv_domid;
>  
>  /* Map the kernel's xenstore page. */
>  void *xenbus_map(void);
> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
> index 5bf16e8..ba9a5ef 100644
> --- a/tools/xenstore/xenstored_domain.c
> +++ b/tools/xenstore/xenstored_domain.c
> @@ -241,7 +241,7 @@ bool domain_can_read(struct connection *conn)
>  
>  bool domain_is_unprivileged(struct connection *conn)
>  {
> -	return (conn && conn->domain && conn->domain->domid != 0);
> +	return (conn && conn->domain && conn->domain->domid != 0 && conn->domain->domid != priv_domid);

Is it deliberate / desirable that both dom0 and domPRIV are privileged
when this option is used? Or should it be just one or the other, which
is equivalent to removing the domid!=0 check since priv_domid defaults
to 0.

Ian.

>  }
>  
>  bool domain_can_write(struct connection *conn)

Re: [PATCH 18/18] xenstored: add --priv-domid parameter

[Daniel De Graaf]

On 01/12/2012 05:20 AM, Ian Campbell wrote:
> On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
>> This parameter identifies an alternative service domain which has
>> superuser access to the xenstore database, which is currently required
>> to set up a new domain's xenstore entries.
>>
>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>> ---
>>  tools/xenstore/xenstored_core.c   |    5 +++++
>>  tools/xenstore/xenstored_core.h   |    1 +
>>  tools/xenstore/xenstored_domain.c |    2 +-
>>  3 files changed, 7 insertions(+), 1 deletions(-)
>>
>> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
>> index 65ceb2c..4437c8d 100644
>> --- a/tools/xenstore/xenstored_core.c
>> +++ b/tools/xenstore/xenstored_core.c
>> @@ -1777,6 +1777,7 @@ static struct option options[] = {
>>  	{ "event", 1, NULL, 'e' },
>>  	{ "help", 0, NULL, 'H' },
>>  	{ "no-fork", 0, NULL, 'N' },
>> +	{ "priv-domid", 1, NULL, 'p' },
>>  	{ "output-pid", 0, NULL, 'P' },
>>  	{ "entry-size", 1, NULL, 'S' },
>>  	{ "trace-file", 1, NULL, 'T' },
>> @@ -1789,6 +1790,7 @@ static struct option options[] = {
>>  
>>  extern void dump_conn(struct connection *conn); 
>>  int dom0_event = 0;
>> +int priv_domid = 0;
>>  
>>  int main(int argc, char *argv[])
>>  {
>> @@ -1854,6 +1856,9 @@ int main(int argc, char *argv[])
>>  		case 'e':
>>  			dom0_event = strtol(optarg, NULL, 10);
>>  			break;
>> +		case 'p':
>> +			priv_domid = strtol(optarg, NULL, 10);
>> +			break;
>>  		}
>>  	}
>>  	if (optind != argc)
>> diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
>> index d3040ba..03e2e48 100644
>> --- a/tools/xenstore/xenstored_core.h
>> +++ b/tools/xenstore/xenstored_core.h
>> @@ -169,6 +169,7 @@ void dtrace_io(const struct connection *conn, const struct buffered_data *data,
>>  
>>  extern int event_fd;
>>  extern int dom0_event;
>> +extern int priv_domid;
>>  
>>  /* Map the kernel's xenstore page. */
>>  void *xenbus_map(void);
>> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
>> index 5bf16e8..ba9a5ef 100644
>> --- a/tools/xenstore/xenstored_domain.c
>> +++ b/tools/xenstore/xenstored_domain.c
>> @@ -241,7 +241,7 @@ bool domain_can_read(struct connection *conn)
>>  
>>  bool domain_is_unprivileged(struct connection *conn)
>>  {
>> -	return (conn && conn->domain && conn->domain->domid != 0);
>> +	return (conn && conn->domain && conn->domain->domid != 0 && conn->domain->domid != priv_domid);
> 
> Is it deliberate / desirable that both dom0 and domPRIV are privileged
> when this option is used? Or should it be just one or the other, which
> is equivalent to removing the domid!=0 check since priv_domid defaults
> to 0.
> 
> Ian.
> 

Yes. In the cases where I used this, both dom0 and domPRIV are privileged.
Since dom0 needs to introduce domPRIV, and introduction is privileged, you
can't just have domPRIV be the only privileged domain. The ideal solution is
to have more fine-grained permissions than superuser-or-domU in xenstored,
but that is a different topic from moving xenstored into a stubdom.

-- 
Daniel De Graaf
National Security Agency

[PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

This adds two ioctls to the /dev/xen/xenbus_backend device allowing the
xenbus backend to be started after the kernel has booted. This is
intended to allow dom0 to start another domain to run xenstore.

IOCTL_XENBUS_BACKEND_REMOTE_SETUP requires that the xenstore domain be
started; the domain ID is passed as the ioctl argument. This sets up a
listening event channel and grant reference to xenbus, but does not
start using this interface. It returns the local event channel port.

IOCTL_XENBUS_BACKEND_REMOTE_COMMIT causes the kernel to begin using the
event channel set up in the previous ioctl, reregistering all watches
and deallocating the previous xenstored event channel.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 drivers/xen/xenbus/xenbus_comms.c       |    6 +++
 drivers/xen/xenbus/xenbus_comms.h       |    1 +
 drivers/xen/xenbus/xenbus_dev_backend.c |   57 +++++++++++++++++++++++++++++++
 include/xen/grant_table.h               |    2 +
 include/xen/xenbus_dev.h                |    6 +++
 5 files changed, 72 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
index 2eff7a6..52fe7ad 100644
--- a/drivers/xen/xenbus/xenbus_comms.c
+++ b/drivers/xen/xenbus/xenbus_comms.c
@@ -234,3 +234,9 @@ int xb_init_comms(void)
 
 	return 0;
 }
+
+void xb_deinit_comms(void)
+{
+	unbind_from_irqhandler(xenbus_irq, &xb_waitq);
+	xenbus_irq = 0;
+}
diff --git a/drivers/xen/xenbus/xenbus_comms.h b/drivers/xen/xenbus/xenbus_comms.h
index 6e42800..c8abd3b 100644
--- a/drivers/xen/xenbus/xenbus_comms.h
+++ b/drivers/xen/xenbus/xenbus_comms.h
@@ -35,6 +35,7 @@
 
 int xs_init(void);
 int xb_init_comms(void);
+void xb_deinit_comms(void);
 
 /* Low level routines. */
 int xb_write(const void *data, unsigned len);
diff --git a/drivers/xen/xenbus/xenbus_dev_backend.c b/drivers/xen/xenbus/xenbus_dev_backend.c
index 3d3be78..4138ba2 100644
--- a/drivers/xen/xenbus/xenbus_dev_backend.c
+++ b/drivers/xen/xenbus/xenbus_dev_backend.c
@@ -8,7 +8,11 @@
 
 #include <xen/xen.h>
 #include <xen/page.h>
+#include <xen/xenbus.h>
 #include <xen/xenbus_dev.h>
+#include <xen/grant_table.h>
+#include <xen/events.h>
+#include <asm/xen/hypervisor.h>
 
 #include "xenbus_comms.h"
 
@@ -22,6 +26,53 @@ static int xenbus_backend_open(struct inode *inode, struct file *filp)
 	return nonseekable_open(inode, filp);
 }
 
+static int pending_xsd_port;
+
+static long xenbus_backend_remote_setup(domid_t domid)
+{
+	struct evtchn_alloc_unbound arg;
+	int err;
+
+	gnttab_grant_foreign_access_ref(GNTTAB_RESERVED_XENSTORE, domid,
+			virt_to_mfn(xen_store_interface), 0 /* writable */);
+
+	if (pending_xsd_port) {
+		struct evtchn_close close;
+		close.port = pending_xsd_port;
+		err = HYPERVISOR_event_channel_op(EVTCHNOP_close, &close);
+		WARN_ON(err);
+	}
+
+	arg.dom = DOMID_SELF;
+	arg.remote_dom = domid;
+
+	err = HYPERVISOR_event_channel_op(EVTCHNOP_alloc_unbound, &arg);
+	if (err)
+		return err;
+
+	pending_xsd_port = arg.port;
+
+	return arg.port;
+}
+
+static long xenbus_backend_remote_commit(unsigned long data)
+{
+	if (!pending_xsd_port)
+		return -EINVAL;
+
+	xs_suspend();
+
+	if (xen_store_evtchn > 0)
+		xb_deinit_comms();
+
+	xen_store_evtchn = pending_xsd_port;
+	pending_xsd_port = 0;
+
+	xs_resume();
+
+	return 0;
+}
+
 static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned long data)
 {
 	if (!capable(CAP_SYS_ADMIN))
@@ -33,6 +84,12 @@ static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned l
 				return xen_store_evtchn;
 			return -ENODEV;
 
+		case IOCTL_XENBUS_BACKEND_REMOTE_SETUP:
+			return xenbus_backend_remote_setup(data);
+
+		case IOCTL_XENBUS_BACKEND_REMOTE_COMMIT:
+			return xenbus_backend_remote_commit(data);
+
 		default:
 			return -ENOTTY;
 	}
diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
index 15f8a00..11e27c3 100644
--- a/include/xen/grant_table.h
+++ b/include/xen/grant_table.h
@@ -46,6 +46,8 @@
 
 #include <xen/features.h>
 
+#define GNTTAB_RESERVED_XENSTORE 1
+
 /* NR_GRANT_FRAMES must be less than or equal to that configured in Xen */
 #define NR_GRANT_FRAMES 4
 
diff --git a/include/xen/xenbus_dev.h b/include/xen/xenbus_dev.h
index ac5f0fe..24d5028 100644
--- a/include/xen/xenbus_dev.h
+++ b/include/xen/xenbus_dev.h
@@ -38,4 +38,10 @@
 #define IOCTL_XENBUS_BACKEND_EVTCHN			\
 	_IOC(_IOC_NONE, 'B', 0, 0)
 
+#define IOCTL_XENBUS_BACKEND_REMOTE_SETUP		\
+	_IOC(_IOC_NONE, 'B', 1, 0)
+
+#define IOCTL_XENBUS_BACKEND_REMOTE_COMMIT		\
+	_IOC(_IOC_NONE, 'B', 2, 0)
+
 #endif /* __LINUX_XEN_XENBUS_DEV_H__ */
-- 
1.7.7.5

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Jan Beulich]

>>> On 11.01.12 at 18:22, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> This adds two ioctls to the /dev/xen/xenbus_backend device allowing the
> xenbus backend to be started after the kernel has booted. This is
> intended to allow dom0 to start another domain to run xenstore.
> 
> IOCTL_XENBUS_BACKEND_REMOTE_SETUP requires that the xenstore domain be
> started; the domain ID is passed as the ioctl argument. This sets up a
> listening event channel and grant reference to xenbus, but does not
> start using this interface. It returns the local event channel port.

Any chance to get at least the setup part matched with the
legacy Linux implementation (defining IOCTL_XENBUS_ALLOC)?

Jan

> IOCTL_XENBUS_BACKEND_REMOTE_COMMIT causes the kernel to begin using the
> event channel set up in the previous ioctl, reregistering all watches
> and deallocating the previous xenstored event channel.
> 
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  drivers/xen/xenbus/xenbus_comms.c       |    6 +++
>  drivers/xen/xenbus/xenbus_comms.h       |    1 +
>  drivers/xen/xenbus/xenbus_dev_backend.c |   57 
> +++++++++++++++++++++++++++++++
>  include/xen/grant_table.h               |    2 +
>  include/xen/xenbus_dev.h                |    6 +++
>  5 files changed, 72 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/xen/xenbus/xenbus_comms.c 
> b/drivers/xen/xenbus/xenbus_comms.c
> index 2eff7a6..52fe7ad 100644
> --- a/drivers/xen/xenbus/xenbus_comms.c
> +++ b/drivers/xen/xenbus/xenbus_comms.c
> @@ -234,3 +234,9 @@ int xb_init_comms(void)
>  
>  	return 0;
>  }
> +
> +void xb_deinit_comms(void)
> +{
> +	unbind_from_irqhandler(xenbus_irq, &xb_waitq);
> +	xenbus_irq = 0;
> +}
> diff --git a/drivers/xen/xenbus/xenbus_comms.h 
> b/drivers/xen/xenbus/xenbus_comms.h
> index 6e42800..c8abd3b 100644
> --- a/drivers/xen/xenbus/xenbus_comms.h
> +++ b/drivers/xen/xenbus/xenbus_comms.h
> @@ -35,6 +35,7 @@
>  
>  int xs_init(void);
>  int xb_init_comms(void);
> +void xb_deinit_comms(void);
>  
>  /* Low level routines. */
>  int xb_write(const void *data, unsigned len);
> diff --git a/drivers/xen/xenbus/xenbus_dev_backend.c 
> b/drivers/xen/xenbus/xenbus_dev_backend.c
> index 3d3be78..4138ba2 100644
> --- a/drivers/xen/xenbus/xenbus_dev_backend.c
> +++ b/drivers/xen/xenbus/xenbus_dev_backend.c
> @@ -8,7 +8,11 @@
>  
>  #include <xen/xen.h>
>  #include <xen/page.h>
> +#include <xen/xenbus.h>
>  #include <xen/xenbus_dev.h>
> +#include <xen/grant_table.h>
> +#include <xen/events.h>
> +#include <asm/xen/hypervisor.h>
>  
>  #include "xenbus_comms.h"
>  
> @@ -22,6 +26,53 @@ static int xenbus_backend_open(struct inode *inode, struct 
> file *filp)
>  	return nonseekable_open(inode, filp);
>  }
>  
> +static int pending_xsd_port;
> +
> +static long xenbus_backend_remote_setup(domid_t domid)
> +{
> +	struct evtchn_alloc_unbound arg;
> +	int err;
> +
> +	gnttab_grant_foreign_access_ref(GNTTAB_RESERVED_XENSTORE, domid,
> +			virt_to_mfn(xen_store_interface), 0 /* writable */);
> +
> +	if (pending_xsd_port) {
> +		struct evtchn_close close;
> +		close.port = pending_xsd_port;
> +		err = HYPERVISOR_event_channel_op(EVTCHNOP_close, &close);
> +		WARN_ON(err);
> +	}
> +
> +	arg.dom = DOMID_SELF;
> +	arg.remote_dom = domid;
> +
> +	err = HYPERVISOR_event_channel_op(EVTCHNOP_alloc_unbound, &arg);
> +	if (err)
> +		return err;
> +
> +	pending_xsd_port = arg.port;
> +
> +	return arg.port;
> +}
> +
> +static long xenbus_backend_remote_commit(unsigned long data)
> +{
> +	if (!pending_xsd_port)
> +		return -EINVAL;
> +
> +	xs_suspend();
> +
> +	if (xen_store_evtchn > 0)
> +		xb_deinit_comms();
> +
> +	xen_store_evtchn = pending_xsd_port;
> +	pending_xsd_port = 0;
> +
> +	xs_resume();
> +
> +	return 0;
> +}
> +
>  static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, 
> unsigned long data)
>  {
>  	if (!capable(CAP_SYS_ADMIN))
> @@ -33,6 +84,12 @@ static long xenbus_backend_ioctl(struct file *file, 
> unsigned int cmd, unsigned l
>  				return xen_store_evtchn;
>  			return -ENODEV;
>  
> +		case IOCTL_XENBUS_BACKEND_REMOTE_SETUP:
> +			return xenbus_backend_remote_setup(data);
> +
> +		case IOCTL_XENBUS_BACKEND_REMOTE_COMMIT:
> +			return xenbus_backend_remote_commit(data);
> +
>  		default:
>  			return -ENOTTY;
>  	}
> diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
> index 15f8a00..11e27c3 100644
> --- a/include/xen/grant_table.h
> +++ b/include/xen/grant_table.h
> @@ -46,6 +46,8 @@
>  
>  #include <xen/features.h>
>  
> +#define GNTTAB_RESERVED_XENSTORE 1
> +
>  /* NR_GRANT_FRAMES must be less than or equal to that configured in Xen */
>  #define NR_GRANT_FRAMES 4
>  
> diff --git a/include/xen/xenbus_dev.h b/include/xen/xenbus_dev.h
> index ac5f0fe..24d5028 100644
> --- a/include/xen/xenbus_dev.h
> +++ b/include/xen/xenbus_dev.h
> @@ -38,4 +38,10 @@
>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>  	_IOC(_IOC_NONE, 'B', 0, 0)
>  
> +#define IOCTL_XENBUS_BACKEND_REMOTE_SETUP		\
> +	_IOC(_IOC_NONE, 'B', 1, 0)
> +
> +#define IOCTL_XENBUS_BACKEND_REMOTE_COMMIT		\
> +	_IOC(_IOC_NONE, 'B', 2, 0)
> +
>  #endif /* __LINUX_XEN_XENBUS_DEV_H__ */
> -- 
> 1.7.7.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com 
> http://lists.xensource.com/xen-devel 

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

On 01/12/2012 03:59 AM, Jan Beulich wrote:
>>>> On 11.01.12 at 18:22, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>> This adds two ioctls to the /dev/xen/xenbus_backend device allowing the
>> xenbus backend to be started after the kernel has booted. This is
>> intended to allow dom0 to start another domain to run xenstore.
>>
>> IOCTL_XENBUS_BACKEND_REMOTE_SETUP requires that the xenstore domain be
>> started; the domain ID is passed as the ioctl argument. This sets up a
>> listening event channel and grant reference to xenbus, but does not
>> start using this interface. It returns the local event channel port.
> 
> Any chance to get at least the setup part matched with the
> legacy Linux implementation (defining IOCTL_XENBUS_ALLOC)?
> 
> Jan

>From what I can tell, the legacy ioctl cannot restore watches because the
return value must be used to set up xenstore communication before they can
be used.

Or were you just asking if IOCTL_XENBUS_BACKEND_REMOTE_SETUP's parameter
can be changed to xenbus_alloc_t?  That structure could be populated in the
same way as the legacy implementation, but it wouldn't give any more
information than the current ioctl does.

> 
>> IOCTL_XENBUS_BACKEND_REMOTE_COMMIT causes the kernel to begin using the
>> event channel set up in the previous ioctl, reregistering all watches
>> and deallocating the previous xenstored event channel.
>>
>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>> ---
[...]

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Jan Beulich]

>>> On 12.01.12 at 16:28, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> On 01/12/2012 03:59 AM, Jan Beulich wrote:
>>>>> On 11.01.12 at 18:22, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>> This adds two ioctls to the /dev/xen/xenbus_backend device allowing the
>>> xenbus backend to be started after the kernel has booted. This is
>>> intended to allow dom0 to start another domain to run xenstore.
>>>
>>> IOCTL_XENBUS_BACKEND_REMOTE_SETUP requires that the xenstore domain be
>>> started; the domain ID is passed as the ioctl argument. This sets up a
>>> listening event channel and grant reference to xenbus, but does not
>>> start using this interface. It returns the local event channel port.
>> 
>> Any chance to get at least the setup part matched with the
>> legacy Linux implementation (defining IOCTL_XENBUS_ALLOC)?
>> 
>> Jan
> 
> From what I can tell, the legacy ioctl cannot restore watches because the
> return value must be used to set up xenstore communication before they can
> be used.

Then I must have missed something. But this was the only kernel side
patch, wasn't it?

Is the intended behavior then to have a Dom0-based xenstored
starting first, then do a brain transfer to that running in a stub
domain? That certainly wasn't the behavior of what the legacy
Linux implementation was intended for...

> Or were you just asking if IOCTL_XENBUS_BACKEND_REMOTE_SETUP's parameter
> can be changed to xenbus_alloc_t?  That structure could be populated in the
> same way as the legacy implementation, but it wouldn't give any more
> information than the current ioctl does.

No, if it functionally differs, then we really should just care that
the ioctl numbers don't collide.

Jan

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

On 01/12/2012 10:40 AM, Jan Beulich wrote:
>>>> On 12.01.12 at 16:28, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>> On 01/12/2012 03:59 AM, Jan Beulich wrote:
>>>>>> On 11.01.12 at 18:22, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>>> This adds two ioctls to the /dev/xen/xenbus_backend device allowing the
>>>> xenbus backend to be started after the kernel has booted. This is
>>>> intended to allow dom0 to start another domain to run xenstore.
>>>>
>>>> IOCTL_XENBUS_BACKEND_REMOTE_SETUP requires that the xenstore domain be
>>>> started; the domain ID is passed as the ioctl argument. This sets up a
>>>> listening event channel and grant reference to xenbus, but does not
>>>> start using this interface. It returns the local event channel port.
>>>
>>> Any chance to get at least the setup part matched with the
>>> legacy Linux implementation (defining IOCTL_XENBUS_ALLOC)?
>>>
>>> Jan
>>
>> From what I can tell, the legacy ioctl cannot restore watches because the
>> return value must be used to set up xenstore communication before they can
>> be used.
> 
> Then I must have missed something. But this was the only kernel side
> patch, wasn't it?
> 
> Is the intended behavior then to have a Dom0-based xenstored
> starting first, then do a brain transfer to that running in a stub
> domain? That certainly wasn't the behavior of what the legacy
> Linux implementation was intended for...
> 
>> Or were you just asking if IOCTL_XENBUS_BACKEND_REMOTE_SETUP's parameter
>> can be changed to xenbus_alloc_t?  That structure could be populated in the
>> same way as the legacy implementation, but it wouldn't give any more
>> information than the current ioctl does.
> 
> No, if it functionally differs, then we really should just care that
> the ioctl numbers don't collide.
> 
> Jan
> 

I think this depends on how the xenstore domain is constructed: if it is
started instead of the dom0-based xenstored (not the current method, but
converting init-xenstored to create the domain itself should make this
possible) then the setup/commit split of the ioctl is not needed. In that
case the older ioctl name/interface can be preserved (although the grant
field will always be populated with 1).

-- 
Daniel De Graaf
National Security Agency

Re: [RFC PATCH 0/18] Xenstore stub domain

[Ian Campbell]

On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> This patch series allows xenstored to run in a stub domian started by
> dom0. It is based on a patch series posted by Alex Zeffertt in 2009 -
> http://old-list-archives.xen.org/archives/html/xen-devel/2009-03/msg01488.html
> 
> 
> A domain configuration for starting xenstored looks like:
> 
> kernel='/home/daniel/xen/stubdom/mini-os-x86_64-xenstore/mini-os'
> extra=''
> memory=50
> name='xenstore'
> 
> Once xenstore is started, "xenstore_dom=1" needs to be added to other
> domain's configurations in order to set up the xenstore connection to
> domain 1.

The domid should be written to xenstore so that the toolstack can pull
it back out as necessary.

> The following program handles post-creation parts of xenstored. To use
> it, run "xl create -p xenstore" and then "init-xenstore $domid". The
> running xenstored must be stopped to prevent xl using the UNIX sockets,
> and xenconsoled needs to be restarted after switching xenstores.

So the model is that you start a normal xenstored process in dom0 and
then use it to start the stub-xenstore before switching over?

How does that work wrt watches which are already registered, e.g. by
backend drivers?

I had imagined a init-xenstore.c which actually built the domain using
libxc directly from a mostly hardcoded configuration as well as
performing the necessary IOCTLs to get around the handover issue.

Ian,

> 
> /* init-xenstore.c: link with -lxenctrl */
> 
> #include <fcntl.h>
> #include <stdio.h>
> #include <string.h>
> #include <stdint.h>
> #include <stdlib.h>
> #include <sys/ioctl.h>
> #include <sys/mman.h>
> 
> #define __XEN_TOOLS__
> #include <xen/domctl.h>
> #include "xenctrl.h"
> 
> #define IOCTL_XENBUS_BACKEND_SETUP _IOC(_IOC_NONE, 'B', 1, 0)
> #define IOCTL_XENBUS_BACKEND_COMMIT _IOC(_IOC_NONE, 'B', 2, 0)
> 
> static void set_virq(int domid, int virq)
> {
> 	struct xen_domctl command;
> 	xc_interface *xch;
> 
> 	xch = xc_interface_open(NULL, NULL, 0);
> 
> 	memset(&command, 0, sizeof(command)); 
> 	command.cmd               = XEN_DOMCTL_set_virq_handler;
> 	command.interface_version = XEN_DOMCTL_INTERFACE_VERSION;
> 	command.domain            = domid;
> 	command.u.set_virq_handler.virq = virq;
> 	xc_domctl(xch, &command);
> 	xc_interface_close(xch);
> }
> 
> int main(int argc, char** argv)
> {
> 	char buf[512];
> 	int domid = atoi(argv[1]);
> 
> 	set_virq(domid, VIRQ_DOM_EXC);
> 
> 	int fd = open("/dev/xen/xenbus_backend", O_RDWR);
> 	void *map = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
> 	int rv = ioctl(fd, IOCTL_XENBUS_BACKEND_SETUP, domid);
> 	*(uint16_t*)(map + 0x810) = rv;
> 	snprintf(buf, 512, "xl unpause %d", domid);
> 	system(buf);
> 	ioctl(fd, IOCTL_XENBUS_BACKEND_COMMIT, 0);
> 	return 0;
> }
> 
> -------------------------------------------------
> 
> Dom0 kernel changes:
>     [PATCH] xenbus: Add support for xenbus backend in stub domain
> 
> This is based on the new /dev/xen devices introduced in Linux 3.3.
> 
> Hypervisor changes:
>     [PATCH 01/18] xen: reinstate previously unused
>     [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to
>     [PATCH 03/18] xsm: allow use of XEN_DOMCTL_getdomaininfo by
>     [PATCH 04/18] xen: Preserve reserved grant entries when switching
> 
> Patch 1 & 4 are required for setting up grant entries in new domains.
> Patch 2 & 3 allow xenstored to run in an unprivileged domain. This
> currently requires XSM to be enabled to avoid allowing all domUs access
> to XEN_DOMCTL_getdomaininfo, so the patch only allows this hypercall if
> XSM is being compiled in.
> 
> Toolstack changes:
>     [PATCH 05/18] tools/libxl: Add xenstore and console backend domain
>     [PATCH 06/18] lib{xc,xl}: Seed grant tables with xenstore and
> 
> These patches populate two of the eight reserved grant entries in new
> domains with the xenstore and console shared pages, which is required
> if xenstored is not run in a privileged domain.
> 
> Minios and xenstored:
>     [PATCH 07/18] mini-os: avoid crash if no console is provided
>     [PATCH 08/18] mini-os: avoid crash if no xenstore is provided
>     [PATCH 09/18] mini-os: remove per-fd evtchn limit
>     [PATCH 10/18] xenstored: use grant references instead of
>     [PATCH 11/18] xenstored: add NO_SOCKETS compilation option
>     [PATCH 12/18] xenstored support for in-memory rather than FS based
>     [PATCH 13/18] xenstored: support running in minios stubdom
>     [PATCH 14/18] xenstored: always use xc_gnttab_munmap in stubdom
>     [PATCH 15/18] xenstored: add --event parameter for bootstrapping
>     [PATCH 16/18] xenstored: pull dom0 event port from shared page
>     [PATCH 17/18] xenstored: use domain_is_unprivileged instead of
>     [PATCH 18/18] xenstored: add --priv-domid parameter
> 
> Support for running in a stub domain
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [RFC PATCH 0/18] Xenstore stub domain

[Ian Campbell]

On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
> This patch series allows xenstored to run in a stub domian started by
> dom0. It is based on a patch series posted by Alex Zeffertt in 2009 -
> http://old-list-archives.xen.org/archives/html/xen-devel/2009-03/msg01488.html

BTW did you come across any patches for stub-oxenstored while you were
doing this? I thought there had been such at some point in the past.

Ian.

> 
> 
> A domain configuration for starting xenstored looks like:
> 
> kernel='/home/daniel/xen/stubdom/mini-os-x86_64-xenstore/mini-os'
> extra=''
> memory=50
> name='xenstore'
> 
> Once xenstore is started, "xenstore_dom=1" needs to be added to other
> domain's configurations in order to set up the xenstore connection to
> domain 1.
> 
> The following program handles post-creation parts of xenstored. To use
> it, run "xl create -p xenstore" and then "init-xenstore $domid". The
> running xenstored must be stopped to prevent xl using the UNIX sockets,
> and xenconsoled needs to be restarted after switching xenstores.
> 
> /* init-xenstore.c: link with -lxenctrl */
> 
> #include <fcntl.h>
> #include <stdio.h>
> #include <string.h>
> #include <stdint.h>
> #include <stdlib.h>
> #include <sys/ioctl.h>
> #include <sys/mman.h>
> 
> #define __XEN_TOOLS__
> #include <xen/domctl.h>
> #include "xenctrl.h"
> 
> #define IOCTL_XENBUS_BACKEND_SETUP _IOC(_IOC_NONE, 'B', 1, 0)
> #define IOCTL_XENBUS_BACKEND_COMMIT _IOC(_IOC_NONE, 'B', 2, 0)
> 
> static void set_virq(int domid, int virq)
> {
> 	struct xen_domctl command;
> 	xc_interface *xch;
> 
> 	xch = xc_interface_open(NULL, NULL, 0);
> 
> 	memset(&command, 0, sizeof(command)); 
> 	command.cmd               = XEN_DOMCTL_set_virq_handler;
> 	command.interface_version = XEN_DOMCTL_INTERFACE_VERSION;
> 	command.domain            = domid;
> 	command.u.set_virq_handler.virq = virq;
> 	xc_domctl(xch, &command);
> 	xc_interface_close(xch);
> }
> 
> int main(int argc, char** argv)
> {
> 	char buf[512];
> 	int domid = atoi(argv[1]);
> 
> 	set_virq(domid, VIRQ_DOM_EXC);
> 
> 	int fd = open("/dev/xen/xenbus_backend", O_RDWR);
> 	void *map = mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
> 	int rv = ioctl(fd, IOCTL_XENBUS_BACKEND_SETUP, domid);
> 	*(uint16_t*)(map + 0x810) = rv;
> 	snprintf(buf, 512, "xl unpause %d", domid);
> 	system(buf);
> 	ioctl(fd, IOCTL_XENBUS_BACKEND_COMMIT, 0);
> 	return 0;
> }
> 
> -------------------------------------------------
> 
> Dom0 kernel changes:
>     [PATCH] xenbus: Add support for xenbus backend in stub domain
> 
> This is based on the new /dev/xen devices introduced in Linux 3.3.
> 
> Hypervisor changes:
>     [PATCH 01/18] xen: reinstate previously unused
>     [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to
>     [PATCH 03/18] xsm: allow use of XEN_DOMCTL_getdomaininfo by
>     [PATCH 04/18] xen: Preserve reserved grant entries when switching
> 
> Patch 1 & 4 are required for setting up grant entries in new domains.
> Patch 2 & 3 allow xenstored to run in an unprivileged domain. This
> currently requires XSM to be enabled to avoid allowing all domUs access
> to XEN_DOMCTL_getdomaininfo, so the patch only allows this hypercall if
> XSM is being compiled in.
> 
> Toolstack changes:
>     [PATCH 05/18] tools/libxl: Add xenstore and console backend domain
>     [PATCH 06/18] lib{xc,xl}: Seed grant tables with xenstore and
> 
> These patches populate two of the eight reserved grant entries in new
> domains with the xenstore and console shared pages, which is required
> if xenstored is not run in a privileged domain.
> 
> Minios and xenstored:
>     [PATCH 07/18] mini-os: avoid crash if no console is provided
>     [PATCH 08/18] mini-os: avoid crash if no xenstore is provided
>     [PATCH 09/18] mini-os: remove per-fd evtchn limit
>     [PATCH 10/18] xenstored: use grant references instead of
>     [PATCH 11/18] xenstored: add NO_SOCKETS compilation option
>     [PATCH 12/18] xenstored support for in-memory rather than FS based
>     [PATCH 13/18] xenstored: support running in minios stubdom
>     [PATCH 14/18] xenstored: always use xc_gnttab_munmap in stubdom
>     [PATCH 15/18] xenstored: add --event parameter for bootstrapping
>     [PATCH 16/18] xenstored: pull dom0 event port from shared page
>     [PATCH 17/18] xenstored: use domain_is_unprivileged instead of
>     [PATCH 18/18] xenstored: add --priv-domid parameter
> 
> Support for running in a stub domain
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [RFC PATCH 0/18] Xenstore stub domain

[Daniel De Graaf]

On 01/12/2012 04:57 AM, Ian Campbell wrote:
> On Wed, 2012-01-11 at 17:21 +0000, Daniel De Graaf wrote:
>> This patch series allows xenstored to run in a stub domian started by
>> dom0. It is based on a patch series posted by Alex Zeffertt in 2009 -
>> http://old-list-archives.xen.org/archives/html/xen-devel/2009-03/msg01488.html
> 
> BTW did you come across any patches for stub-oxenstored while you were
> doing this? I thought there had been such at some point in the past.
> 
> Ian.
> 

I haven't searched for similar patches to oxenstored. If they exist,
they would probably need similar changes to use grants instead of
map_foreign_range and support for the bootstrapping. Since I am not
fluent in ocaml, I will leave this for other people to implement :)

Re: [RFC PATCH 0/18] Xenstore stub domain

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 776 bytes --]

On 01/11/12 18:21, Daniel De Graaf wrote:
> This patch series allows xenstored to run in a stub domian started by
> dom0. It is based on a patch series posted by Alex Zeffertt in 2009 -
> http://old-list-archives.xen.org/archives/html/xen-devel/2009-03/msg01488.html
> 

Daniel,

Can you explain what is the rationale for moving the xenstored into a
stubdom? After all, if an attacker is able to compromise the xenstored,
there should be many ways now how to compromise other VMs in the system?
And it shouldn't matter whether the xenstored is in stubdom or whether
in Dom0. E.g. the attacker might redirect the block fronts to us some
false block backends, so that the VMs get compromised fs. One could
probably think of other attacks as well...?

joanna.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [RFC PATCH 0/18] Xenstore stub domain

[Tim Deegan]

At 11:33 +0100 on 12 Jan (1326367997), Joanna Rutkowska wrote:
> Daniel,
> 
> Can you explain what is the rationale for moving the xenstored into a
> stubdom? After all, if an attacker is able to compromise the xenstored,
> there should be many ways now how to compromise other VMs in the system?
> And it shouldn't matter whether the xenstored is in stubdom or whether
> in Dom0. E.g. the attacker might redirect the block fronts to us some
> false block backends, so that the VMs get compromised fs. One could
> probably think of other attacks as well...?

I think the point is to protect xenstore from dom0, not dom0 from
xenstore.  With stub-xenstore and driver domains, only the domain
builder and PCIback need to have any privilege, and they can be moved
out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
http://tjd.phlegethon.org/words/sosp11-xoar.html)

Tim.

On Dom0 disaggregation (was: Re: [RFC PATCH 0/18] Xenstore stub domain)

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 2123 bytes --]

On 01/12/12 11:48, Tim Deegan wrote:
> At 11:33 +0100 on 12 Jan (1326367997), Joanna Rutkowska wrote:
>> Daniel,
>>
>> Can you explain what is the rationale for moving the xenstored into a
>> stubdom? After all, if an attacker is able to compromise the xenstored,
>> there should be many ways now how to compromise other VMs in the system?
>> And it shouldn't matter whether the xenstored is in stubdom or whether
>> in Dom0. E.g. the attacker might redirect the block fronts to us some
>> false block backends, so that the VMs get compromised fs. One could
>> probably think of other attacks as well...?
> 
> I think the point is to protect xenstore from dom0, not dom0 from
> xenstore.  With stub-xenstore and driver domains, only the domain
> builder and PCIback need to have any privilege, and they can be moved
> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
> http://tjd.phlegethon.org/words/sosp11-xoar.html)
> 

In order for this to make sense from security point of view, you would
need to deprivilige Dom0. When considering this task, one should answer
the following questions:

1) Who manages the chipset (MCH)?
2) Who manages the input (keyboard, mouse)
3) Who manages the output (GPU, specifically critical on client systems)

From the security point of view there is no point of isolating the
entities that manage the above between each other, because a compromise
of any of those entities leads to full system compromise (again, #3
applies to client systems).

Now, as you pointed out, we shall probably add another bullet to the
list, which is:

4) the xenstored

(although I'm not 100% if we couldn't somehow deprivilige xenstored).

So, we end up with a conclusion that there is no point separating those
4 functionalists between each other, and so it only make sense to host
them in the same domain. How about we call this domain "Dom0", then? ;)

(Sure, this "new Dom0" doesn't include net, USB, and perhaps even SATA
drivers, but we already have been doing this on Qubes, except for moving
out SATA/blkbackend from Dom0).

joanna.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: On Dom0 disaggregation (was: Re: [RFC PATCH 0/18] Xenstore stub domain)

[Tim Deegan]

At 12:18 +0100 on 12 Jan (1326370728), Joanna Rutkowska wrote:
> On 01/12/12 11:48, Tim Deegan wrote:
> > I think the point is to protect xenstore from dom0, not dom0 from
> > xenstore.  With stub-xenstore and driver domains, only the domain
> > builder and PCIback need to have any privilege, and they can be moved
> > out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
> > http://tjd.phlegethon.org/words/sosp11-xoar.html)
> 
> In order for this to make sense from security point of view, you would
> need to deprivilige Dom0.

Yep.  That's what Xoar did (or, if you prefer, it moved all dom0's
components out and then got rid of it).

> When considering this task, one should answer
> the following questions:
> 
> 1) Who manages the chipset (MCH)?

A driver domain that contains PCIback.  It doesn't need general
privlege (e.g. for scheduler or memory operations).

> 2) Who manages the input (keyboard, mouse)
> 3) Who manages the output (GPU, specifically critical on client systems)
> 
> From the security point of view there is no point of isolating the
> entities that manage the above between each other, because a compromise
> of any of those entities leads to full system compromise (again, #3
> applies to client systems).

#2 is really a client-only thing as well.  Serial console input for
servers is owned directly by the hypervisor.

> Now, as you pointed out, we shall probably add another bullet to the
> list, which is:
> 
> 4) the xenstored
> 
> (although I'm not 100% if we couldn't somehow deprivilige xenstored).

Yes we could (and Xoar did).  IIRC the only reason it needs privilege is
to map the rings and that can be sorted out by having the builder
pre-load a grant entry.

> So, we end up with a conclusion that there is no point separating those
> 4 functionalists between each other, and so it only make sense to host
> them in the same domain. How about we call this domain "Dom0", then? ;)

So you're saying that the PCIback domain (which owns the chipsets) might
as well host Xenstore since either of them could hose the system.
Maybe so.  In Xoar we had two reasons not to do that:
 - in some configurations you could shut that domain down after boot
   (though tbh doing that leads to poor error handling!); and
 - we were doing other hardening of Xenstore that involved breaking it 
   up into more than one VM. 

Cheers,

Tim.

Re: On Dom0 disaggregation

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 3401 bytes --]

On 01/12/12 13:13, Tim Deegan wrote:
> At 12:18 +0100 on 12 Jan (1326370728), Joanna Rutkowska wrote:
>> On 01/12/12 11:48, Tim Deegan wrote:
>>> I think the point is to protect xenstore from dom0, not dom0 from
>>> xenstore.  With stub-xenstore and driver domains, only the domain
>>> builder and PCIback need to have any privilege, and they can be moved
>>> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
>>> http://tjd.phlegethon.org/words/sosp11-xoar.html)
>>
>> In order for this to make sense from security point of view, you would
>> need to deprivilige Dom0.
> 
> Yep.  That's what Xoar did (or, if you prefer, it moved all dom0's
> components out and then got rid of it).
> 
>> When considering this task, one should answer
>> the following questions:
>>
>> 1) Who manages the chipset (MCH)?
> 
> A driver domain that contains PCIback.  It doesn't need general
> privlege (e.g. for scheduler or memory operations).
> 
>> 2) Who manages the input (keyboard, mouse)
>> 3) Who manages the output (GPU, specifically critical on client systems)
>>
>> From the security point of view there is no point of isolating the
>> entities that manage the above between each other, because a compromise
>> of any of those entities leads to full system compromise (again, #3
>> applies to client systems).
> 
> #2 is really a client-only thing as well.  Serial console input for
> servers is owned directly by the hypervisor.
> 
>> Now, as you pointed out, we shall probably add another bullet to the
>> list, which is:
>>
>> 4) the xenstored
>>
>> (although I'm not 100% if we couldn't somehow deprivilige xenstored).
> 
> Yes we could (and Xoar did).  IIRC the only reason it needs privilege is
> to map the rings and that can be sorted out by having the builder
> pre-load a grant entry.
> 
>> So, we end up with a conclusion that there is no point separating those
>> 4 functionalists between each other, and so it only make sense to host
>> them in the same domain. How about we call this domain "Dom0", then? ;)
> 
> So you're saying that the PCIback domain (which owns the chipsets) might
> as well host Xenstore since either of them could hose the system.
> Maybe so.  In Xoar we had two reasons not to do that:
>  - in some configurations you could shut that domain down after boot
>    (though tbh doing that leads to poor error handling!); and
>  - we were doing other hardening of Xenstore that involved breaking it 
>    up into more than one VM. 
> 

But the paper says the PCIBack is destroyed after init -- who manages
the chipset (MCH, ICH, platform power management), then?

Also, if I correctly interpret your architecture, then BlkBack VM must
be trusted, as it has access to the disk, and thus can serve compromised
fs/kernel/initramfs images to VMs, and also modify boot partition to
load compromised Xen on the next boot (unless you use something like
Intel TXT for Xen load). So, what's the reason of separating it from
(the trusted) Xenstored VM?

Unless the kernel/initramfs images (at least for PV domains) come from
some trusted source, not from blkbackend? Then they could contain code
to verify integrity of the fs served from blkbackend. Have you
considered this in your architecture?

Also, if this was a client system, where would you put user input/output
handling?

joanna.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: On Dom0 disaggregation

[Tim Deegan]

At 14:30 +0100 on 12 Jan (1326378636), Joanna Rutkowska wrote:
> > So you're saying that the PCIback domain (which owns the chipsets) might
> > as well host Xenstore since either of them could hose the system.
> > Maybe so.  In Xoar we had two reasons not to do that:
> >  - in some configurations you could shut that domain down after boot
> >    (though tbh doing that leads to poor error handling!); and
> >  - we were doing other hardening of Xenstore that involved breaking it 
> >    up into more than one VM. 
> > 
> 
> But the paper says the PCIBack is destroyed after init -- who manages
> the chipset (MCH, ICH, platform power management), then?

Nobody.  Like I said, it's not ideal. :)

You don't have to destroy PCIBack -- you can keep it around to do those
things.  And if you do, it has a _much_ narrower interface to the rest
of the world than dom0 does.  It has no network access, and no direct
interface to customer (i.e. non-driver) VMs.  A malicious customer VM
must break at least one other system VM before it can attack it.

> Also, if I correctly interpret your architecture, then BlkBack VM must
> be trusted, as it has access to the disk, and thus can serve compromised
> fs/kernel/initramfs images to VMs, and also modify boot partition to
> load compromised Xen on the next boot (unless you use something like
> Intel TXT for Xen load). So, what's the reason of separating it from
> (the trusted) Xenstored VM?

Well, you can have multiple block driver domains, if you have multiple
HBAs, and arrange that two customers' data don't share a blkback.

And there are the usual reasons for driver domains.  For example you're
not tied to a particular OS for all your drivers.  And device-driver
crashes don't take out xenstore.

Trusted boot is orthogonal to the Xoar work.  We didn't go into it in
the paper but it's fairly obvious how it would fit in.  And once you've
got a TPM you could also use vTPMs to protect VMs' disk images.

> Also, if this was a client system, where would you put user input/output
> handling?

I'd probably make a console driver VM that owned the GPU and the USB
controller.  It might make sense for the toolstack (or at least the
parts that compose the display and demux the inputs) to live in that VM
too.  Or maybe it would be better to isolate those drivers; I haven't
given it a lot of thought. 

Cheers,

Tim.

Re: On Dom0 disaggregation

[Mihir Nanavati]

On 2012-01-12, at 5:31 AM, Joanna Rutkowska <joanna@invisiblethingslab.com> wrote:

> On 01/12/12 13:13, Tim Deegan wrote:
>> At 12:18 +0100 on 12 Jan (1326370728), Joanna Rutkowska wrote:
>>> On 01/12/12 11:48, Tim Deegan wrote:
>>>> I think the point is to protect xenstore from dom0, not dom0 from
>>>> xenstore.  With stub-xenstore and driver domains, only the domain
>>>> builder and PCIback need to have any privilege, and they can be moved
>>>> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
>>>> http://tjd.phlegethon.org/words/sosp11-xoar.html)
>>> 
>>> In order for this to make sense from security point of view, you would
>>> need to deprivilige Dom0.
>> 
>> Yep.  That's what Xoar did (or, if you prefer, it moved all dom0's
>> components out and then got rid of it).
>> 
>>> When considering this task, one should answer
>>> the following questions:
>>> 
>>> 1) Who manages the chipset (MCH)?
>> 
>> A driver domain that contains PCIback.  It doesn't need general
>> privlege (e.g. for scheduler or memory operations).
>> 
>>> 2) Who manages the input (keyboard, mouse)
>>> 3) Who manages the output (GPU, specifically critical on client systems)
>>> 
>>> From the security point of view there is no point of isolating the
>>> entities that manage the above between each other, because a compromise
>>> of any of those entities leads to full system compromise (again, #3
>>> applies to client systems).
>> 
>> #2 is really a client-only thing as well.  Serial console input for
>> servers is owned directly by the hypervisor.
>> 
>>> Now, as you pointed out, we shall probably add another bullet to the
>>> list, which is:
>>> 
>>> 4) the xenstored
>>> 
>>> (although I'm not 100% if we couldn't somehow deprivilige xenstored).
>> 
>> Yes we could (and Xoar did).  IIRC the only reason it needs privilege is
>> to map the rings and that can be sorted out by having the builder
>> pre-load a grant entry.
>> 
>>> So, we end up with a conclusion that there is no point separating those
>>> 4 functionalists between each other, and so it only make sense to host
>>> them in the same domain. How about we call this domain "Dom0", then? ;)
>> 
>> So you're saying that the PCIback domain (which owns the chipsets) might
>> as well host Xenstore since either of them could hose the system.
>> Maybe so.  In Xoar we had two reasons not to do that:
>> - in some configurations you could shut that domain down after boot
>>   (though tbh doing that leads to poor error handling!); and
>> - we were doing other hardening of Xenstore that involved breaking it 
>>   up into more than one VM. 
>> 
> 
> But the paper says the PCIBack is destroyed after init -- who manages
> the chipset (MCH, ICH, platform power management), then?
> 
> Also, if I correctly interpret your architecture, then BlkBack VM must
> be trusted, as it has access to the disk, and thus can serve compromised
> fs/kernel/initramfs images to VMs, and also modify boot partition to
> load compromised Xen on the next boot (unless you use something like
> Intel TXT for Xen load). So, what's the reason of separating it from
> (the trusted) Xenstored VM?
> 
> Unless the kernel/initramfs images (at least for PV domains) come from
> some trusted source, not from blkbackend? Then they could contain code
> to verify integrity of the fs served from blkbackend. Have you
> considered this in your architecture?

Indeed. The domain builder only builds known good kernels, currently which are served from its ramdisk. There is an alternative architecture where it's served by a special storage domain via shared memory too.

You're right about the block backend needing to be trusted to some extent; it's only a matter of reducing who needs to trust it. On a system with multiple disk controllers, the one serving Xen could be split from the one serving the VMs. With only a single controller, you'd probably use TXT.


> 
> Also, if this was a client system, where would you put user input/output
> handling?

I'm afraid I'm not sure I know exactly what you mean here...

~M

> 
> joanna.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [RFC PATCH 0/18] Xenstore stub domain

[Ian Campbell]

On Thu, 2012-01-12 at 10:48 +0000, Tim Deegan wrote:
> At 11:33 +0100 on 12 Jan (1326367997), Joanna Rutkowska wrote:
> > Daniel,
> > 
> > Can you explain what is the rationale for moving the xenstored into a
> > stubdom? After all, if an attacker is able to compromise the xenstored,
> > there should be many ways now how to compromise other VMs in the system?
> > And it shouldn't matter whether the xenstored is in stubdom or whether
> > in Dom0. E.g. the attacker might redirect the block fronts to us some
> > false block backends, so that the VMs get compromised fs. One could
> > probably think of other attacks as well...?
> 
> I think the point is to protect xenstore from dom0, not dom0 from
> xenstore.  With stub-xenstore and driver domains, only the domain
> builder and PCIback need to have any privilege, and they can be moved
> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
> http://tjd.phlegethon.org/words/sosp11-xoar.html)

Also by isolating components you gain the ability to restart them
independently. Since xenstored is one of (the only?) dom0 component
which cannot be trivially restarted so putting it in a separate domain
means you can restart dom0.

Ian.

Re: [RFC PATCH 0/18] Xenstore stub domain

[Vasiliy Tolstov]

2012/1/12 Ian Campbell <Ian.Campbell@citrix.com>:
> On Thu, 2012-01-12 at 10:48 +0000, Tim Deegan wrote:
>> At 11:33 +0100 on 12 Jan (1326367997), Joanna Rutkowska wrote:
>> > Daniel,
>> >
>> > Can you explain what is the rationale for moving the xenstored into a
>> > stubdom? After all, if an attacker is able to compromise the xenstored,
>> > there should be many ways now how to compromise other VMs in the system?
>> > And it shouldn't matter whether the xenstored is in stubdom or whether
>> > in Dom0. E.g. the attacker might redirect the block fronts to us some
>> > false block backends, so that the VMs get compromised fs. One could
>> > probably think of other attacks as well...?
>>
>> I think the point is to protect xenstore from dom0, not dom0 from
>> xenstore.  With stub-xenstore and driver domains, only the domain
>> builder and PCIback need to have any privilege, and they can be moved
>> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
>> http://tjd.phlegethon.org/words/sosp11-xoar.html)
>
> Also by isolating components you gain the ability to restart them
> independently. Since xenstored is one of (the only?) dom0 component
> which cannot be trivially restarted so putting it in a separate domain
> means you can restart dom0.
>
>

Is that possible now (or in some feature) to restart xenstore domain
and not loose watches? Now if i save all contents in xenstore and
restart xenstored i'm loosing all watches and domU's not able to work
correctly...

-- 
Vasiliy Tolstov,
Clodo.ru
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [RFC PATCH 0/18] Xenstore stub domain

[Ian Campbell]

On Thu, 2012-01-12 at 11:33 +0000, Vasiliy Tolstov wrote:
> 2012/1/12 Ian Campbell <Ian.Campbell@citrix.com>:
> > On Thu, 2012-01-12 at 10:48 +0000, Tim Deegan wrote:
> >> At 11:33 +0100 on 12 Jan (1326367997), Joanna Rutkowska wrote:
> >> > Daniel,
> >> >
> >> > Can you explain what is the rationale for moving the xenstored into a
> >> > stubdom? After all, if an attacker is able to compromise the xenstored,
> >> > there should be many ways now how to compromise other VMs in the system?
> >> > And it shouldn't matter whether the xenstored is in stubdom or whether
> >> > in Dom0. E.g. the attacker might redirect the block fronts to us some
> >> > false block backends, so that the VMs get compromised fs. One could
> >> > probably think of other attacks as well...?
> >>
> >> I think the point is to protect xenstore from dom0, not dom0 from
> >> xenstore.  With stub-xenstore and driver domains, only the domain
> >> builder and PCIback need to have any privilege, and they can be moved
> >> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
> >> http://tjd.phlegethon.org/words/sosp11-xoar.html)
> >
> > Also by isolating components you gain the ability to restart them
> > independently. Since xenstored is one of (the only?) dom0 component
> > which cannot be trivially restarted so putting it in a separate domain
> > means you can restart dom0.
> >
> >
> 
> Is that possible now (or in some feature) to restart xenstore domain
> and not loose watches? Now if i save all contents in xenstore and
> restart xenstored i'm loosing all watches and domU's not able to work
> correctly...

I've heard that it is possible to do this with the ocaml xenstored. I
haven't tried it myself.

Ian.

Re: [RFC PATCH 0/18] Xenstore stub domain

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 1349 bytes --]

On 01/12/12 12:27, Ian Campbell wrote:
> On Thu, 2012-01-12 at 10:48 +0000, Tim Deegan wrote:
>> At 11:33 +0100 on 12 Jan (1326367997), Joanna Rutkowska wrote:
>>> Daniel,
>>>
>>> Can you explain what is the rationale for moving the xenstored into a
>>> stubdom? After all, if an attacker is able to compromise the xenstored,
>>> there should be many ways now how to compromise other VMs in the system?
>>> And it shouldn't matter whether the xenstored is in stubdom or whether
>>> in Dom0. E.g. the attacker might redirect the block fronts to us some
>>> false block backends, so that the VMs get compromised fs. One could
>>> probably think of other attacks as well...?
>>
>> I think the point is to protect xenstore from dom0, not dom0 from
>> xenstore.  With stub-xenstore and driver domains, only the domain
>> builder and PCIback need to have any privilege, and they can be moved
>> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
>> http://tjd.phlegethon.org/words/sosp11-xoar.html)
> 
> Also by isolating components you gain the ability to restart them
> independently. Since xenstored is one of (the only?) dom0 component
> which cannot be trivially restarted so putting it in a separate domain
> means you can restart dom0.
> 

But why would anybody want to restart Dom0, in the first place?


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [RFC PATCH 0/18] Xenstore stub domain

[Ian Campbell]

On Thu, 2012-01-12 at 11:35 +0000, Joanna Rutkowska wrote:
> On 01/12/12 12:27, Ian Campbell wrote:
> > On Thu, 2012-01-12 at 10:48 +0000, Tim Deegan wrote:
> >> At 11:33 +0100 on 12 Jan (1326367997), Joanna Rutkowska wrote:
> >>> Daniel,
> >>>
> >>> Can you explain what is the rationale for moving the xenstored into a
> >>> stubdom? After all, if an attacker is able to compromise the xenstored,
> >>> there should be many ways now how to compromise other VMs in the system?
> >>> And it shouldn't matter whether the xenstored is in stubdom or whether
> >>> in Dom0. E.g. the attacker might redirect the block fronts to us some
> >>> false block backends, so that the VMs get compromised fs. One could
> >>> probably think of other attacks as well...?
> >>
> >> I think the point is to protect xenstore from dom0, not dom0 from
> >> xenstore.  With stub-xenstore and driver domains, only the domain
> >> builder and PCIback need to have any privilege, and they can be moved
> >> out of dom0 too (e.g., http://dl.acm.org/citation.cfm?id=1346278 ,
> >> http://tjd.phlegethon.org/words/sosp11-xoar.html)
> > 
> > Also by isolating components you gain the ability to restart them
> > independently. Since xenstored is one of (the only?) dom0 component
> > which cannot be trivially restarted so putting it in a separate domain
> > means you can restart dom0.
> > 
> 
> But why would anybody want to restart Dom0, in the first place?

To patch the kernel?

Ian.

Re: [RFC PATCH 0/18] Xenstore stub domain

[Keir Fraser]

On 12/01/2012 10:33, "Joanna Rutkowska" <joanna@invisiblethingslab.com>
wrote:

> On 01/11/12 18:21, Daniel De Graaf wrote:
>> This patch series allows xenstored to run in a stub domian started by
>> dom0. It is based on a patch series posted by Alex Zeffertt in 2009 -
>> 
http://old-list-archives.xen.org/archives/html/xen-devel/2009-03/msg01488.htm>>
l
>> 
> 
> Daniel,
> 
> Can you explain what is the rationale for moving the xenstored into a
> stubdom? After all, if an attacker is able to compromise the xenstored,
> there should be many ways now how to compromise other VMs in the system?
> And it shouldn't matter whether the xenstored is in stubdom or whether
> in Dom0. E.g. the attacker might redirect the block fronts to us some
> false block backends, so that the VMs get compromised fs. One could
> probably think of other attacks as well...?

As you point out it's a critical component in itself, so I suppose this work
is mainly about isolating it from the big attack surfaces in dom0. It's of
questionable value unless dom0 itself can be deprivileged, or the big attack
surfaces themselves shuffled off into lesser-privileged domains. In a well
locked down dom0 I would say that the biggest attack surfaces are via things
like domain build, save/restore, and qemu, being intrinsic (ie unavoidable)
components of a Xen system which consume complex inputs. We can already do
qemu stubdoms, perhaps with some features missing still. Launching isolated,
de-privileged (eg can only act on the one specified domain),
domain-builder/saver/restorer stubdoms would be an interesting direction
imo. It's easy to be impressed with any disaggregation effort almost for its
own sake, and lose sight of the importance of basic security analysis as a
starting point.

 -- Keir

> joanna.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [RFC PATCH 0/18] Xenstore stub domain

[Daniel De Graaf]

On 01/12/2012 05:33 AM, Joanna Rutkowska wrote:
> On 01/11/12 18:21, Daniel De Graaf wrote:
>> This patch series allows xenstored to run in a stub domian started by
>> dom0. It is based on a patch series posted by Alex Zeffertt in 2009 -
>> http://old-list-archives.xen.org/archives/html/xen-devel/2009-03/msg01488.html
>>
> 
> Daniel,
> 
> Can you explain what is the rationale for moving the xenstored into a
> stubdom? After all, if an attacker is able to compromise the xenstored,
> there should be many ways now how to compromise other VMs in the system?
> And it shouldn't matter whether the xenstored is in stubdom or whether
> in Dom0. E.g. the attacker might redirect the block fronts to us some
> false block backends, so that the VMs get compromised fs. One could
> probably think of other attacks as well...?
> 
> joanna.
> 

Splitting xenstored into its own domain (rather than keeping it in dom0)
means that it does not need to be privileged, so a compromise of xenstore
does not automatically give you full access to all other domains on the
system.

While it is possible to attack domains by sending them bad commands from
xenstore (device unplug/domain changes), it is also possible for a guest
VM to detect this and it becomes a denial-of-service instead of a way to
compromise the system. This is most easily done if guests use their own
full-disk encryption and run integrity checks on the unencrypted parts
(kernel/initrd; using a vTPM to unlock the guest-based FDE would work).

This split also prevents xenstored from being attacked from dom0, but this
is currently not as important security-wise since dom0 has superuser access
to the xenstore database. However, it does allow for future changes to
xenstore's security model that do not include a fully-privileged domain.

-- 
Daniel De Graaf
National Security Agency

[PATCH v2 00/18] Xenstore stub domain

[Daniel De Graaf]

Changes from v1:
 - set_virq_handler implemented in libxc
 - added custom domain builder for xenstored
 - xenstore/console domain IDs now pulled from xenstore
 - migration support when using split xenstored (untested, should work)
 - slightly less intrusive NO_SOCKETS xenstored patch
   (still has many ifdefs to avoid pulling in socket headers or symbols)
 - virq handlers removed from dying domain when clearing event channels
 - dummy XSM module restricts getdomaininfo similar to no-XSM case
 - formatting/type fixups
 - partial ioctl compatibility with legacy IOCTL_XENBUS_ALLOC
   (the new device path uses 'B' not 'X' as the base so the ioctl number
   does not match; otherwise, should be compatible).

To start xenstored, run:

tools/xenstore/init-xenstore-domain stubdom/mini-os-x86_64-xenstore/mini-os 20 system_u:system_r:domU_t

This will populate the xenstore domid key /tool/xenstore/domid

----

[PATCH 01/18] xen: reinstate previously unused hypercall
[PATCH 02/18] xen: allow global VIRQ handlers to be delegated to
[PATCH 03/18] xen: use XSM instead of IS_PRIV for getdomaininfo
[PATCH 04/18] xen: Preserve reserved grant entries when switching

[PATCH 05/18] tools/libxl: pull xenstore/console domids from
[PATCH 06/18] lib{xc,xl}: Seed grant tables with xenstore and

[PATCH 07/18] mini-os: avoid crash if no console is provided
[PATCH 08/18] mini-os: avoid crash if no xenstore is provided
[PATCH 09/18] mini-os: remove per-fd evtchn limit
[PATCH 10/18] xenstored: use grant references instead of
[PATCH 11/18] xenstored: add NO_SOCKETS compilation option
[PATCH 12/18] xenstored support for in-memory rather than FS based
[PATCH 13/18] xenstored: support running in minios stubdom
[PATCH 14/18] xenstored: always use xc_gnttab_munmap in stubdom
[PATCH 15/18] xenstored: add --event parameter for bootstrapping
Removed old #16; shared page is no longer used to pass event.
[PATCH 16/18] xenstored: use domain_is_unprivileged instead of
[PATCH 17/18] xenstored: add --priv-domid parameter

[PATCH 18/18] xenstored: Add stub domain builder

[PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

This patch reinstates the XENMEM_remove_from_physmap hypercall
which was removed in 19041:ee62aaafff46 because it was not used.

However, is now needed in order to support xenstored stub domains.
The xenstored stub domain is not priviliged like dom0 and so cannot
unilaterally map the xenbus page of other guests into it's address
space.  Therefore, before creating a domU the domain builder needs to
seed its grant table with a grant ref allowing the xenstored stub
domain to access the new domU's xenbus page.

At present domU's do not start with their grant table mapped.
Instead it gets mapped when the guest requests a grant table from
the hypervisor.

In order to seed the grant table, the domain builder first needs to
map it into dom0 address space.  But the hypercall to do this
requires a gpfn (guest pfn), which is an mfn for PV guest, but a pfn
for HVM guests.  Therfore, in order to seed the grant table of an
HVM guest, dom0 needs to *temporarily* map it into the guest's
"physical" address space.

Hence the need to reinstate the XENMEM_remove_from_physmap hypercall.

Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 xen/arch/ia64/xen/mm.c          |   34 ++++++++++++++++++++++++++++++++++
 xen/arch/x86/mm.c               |   35 +++++++++++++++++++++++++++++++++++
 xen/arch/x86/x86_64/compat/mm.c |   14 ++++++++++++++
 xen/include/public/memory.h     |   16 ++++++++++++++++
 xen/include/xlat.lst            |    1 +
 xen/include/xsm/xsm.h           |    6 ++++++
 xen/xsm/dummy.c                 |    6 ++++++
 xen/xsm/flask/hooks.c           |    6 ++++++
 8 files changed, 118 insertions(+), 0 deletions(-)

diff --git a/xen/arch/ia64/xen/mm.c b/xen/arch/ia64/xen/mm.c
index 84f6a61..a40f96a 100644
--- a/xen/arch/ia64/xen/mm.c
+++ b/xen/arch/ia64/xen/mm.c
@@ -3448,6 +3448,40 @@ arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
         break;
     }
 
+    case XENMEM_remove_from_physmap:
+    {
+        struct xen_remove_from_physmap xrfp;
+        unsigned long mfn;
+        struct domain *d;
+
+        if ( copy_from_guest(&xrfp, arg, 1) )
+            return -EFAULT;
+
+        rc = rcu_lock_target_domain_by_id(xrfp.domid, &d);
+        if ( rc != 0 )
+            return rc;
+
+        if ( xsm_remove_from_physmap(current->domain, d) )
+        {
+            rcu_unlock_domain(d);
+            return -EPERM;
+        }
+
+        domain_lock(d);
+
+        mfn = gmfn_to_mfn(d, xrfp.gpfn);
+
+        if ( mfn_valid(mfn) )
+            guest_physmap_remove_page(d, xrfp.gpfn, mfn, 0);
+
+        domain_unlock(d);
+
+        rcu_unlock_domain(d);
+
+        break;
+    }
+
+
     case XENMEM_machine_memory_map:
     {
         struct xen_memory_map memmap;
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 1f996e0..39cc3c0 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -4871,6 +4871,41 @@ long arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
         return rc;
     }
 
+    case XENMEM_remove_from_physmap:
+    {
+        struct xen_remove_from_physmap xrfp;
+        unsigned long mfn;
+        struct domain *d;
+
+        if ( copy_from_guest(&xrfp, arg, 1) )
+            return -EFAULT;
+
+        rc = rcu_lock_target_domain_by_id(xrfp.domid, &d);
+        if ( rc != 0 )
+            return rc;
+
+        if ( xsm_remove_from_physmap(current->domain, d) )
+        {
+            rcu_unlock_domain(d);
+            return -EPERM;
+        }
+
+        domain_lock(d);
+
+        mfn = get_gfn_untyped(d, xrfp.gpfn);
+
+        if ( mfn_valid(mfn) )
+            guest_physmap_remove_page(d, xrfp.gpfn, mfn, PAGE_ORDER_4K);
+
+        put_gfn(d, xrfp.gpfn);
+
+        domain_unlock(d);
+
+        rcu_unlock_domain(d);
+
+        break;
+    }
+
     case XENMEM_set_memory_map:
     {
         struct xen_foreign_memory_map fmap;
diff --git a/xen/arch/x86/x86_64/compat/mm.c b/xen/arch/x86/x86_64/compat/mm.c
index bea94fe..dde5430 100644
--- a/xen/arch/x86/x86_64/compat/mm.c
+++ b/xen/arch/x86/x86_64/compat/mm.c
@@ -82,6 +82,20 @@ int compat_arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
         break;
     }
 
+    case XENMEM_remove_from_physmap:
+    {
+        struct compat_remove_from_physmap cmp;
+        struct xen_remove_from_physmap *nat = (void *)COMPAT_ARG_XLAT_VIRT_BASE;
+
+        if ( copy_from_guest(&cmp, arg, 1) )
+            return -EFAULT;
+
+        XLAT_remove_from_physmap(nat, &cmp);
+        rc = arch_memory_op(op, guest_handle_from_ptr(nat, void));
+
+        break;
+    }
+
     case XENMEM_set_memory_map:
     {
         struct compat_foreign_memory_map cmp;
diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
index c5b78a8..308deff 100644
--- a/xen/include/public/memory.h
+++ b/xen/include/public/memory.h
@@ -229,6 +229,22 @@ struct xen_add_to_physmap {
 typedef struct xen_add_to_physmap xen_add_to_physmap_t;
 DEFINE_XEN_GUEST_HANDLE(xen_add_to_physmap_t);
 
+/*
+ * Unmaps the page appearing at a particular GPFN from the specified guest's
+ * pseudophysical address space.
+ * arg == addr of xen_remove_from_physmap_t.
+ */
+#define XENMEM_remove_from_physmap      15
+struct xen_remove_from_physmap {
+    /* Which domain to change the mapping for. */
+    domid_t domid;
+
+    /* GPFN of the current mapping of the page. */
+    xen_pfn_t     gpfn;
+};
+typedef struct xen_remove_from_physmap xen_remove_from_physmap_t;
+DEFINE_XEN_GUEST_HANDLE(xen_remove_from_physmap_t);
+
 /*** REMOVED ***/
 /*#define XENMEM_translate_gpfn_list  8*/
 
diff --git a/xen/include/xlat.lst b/xen/include/xlat.lst
index 3d92175..ee1772c 100644
--- a/xen/include/xlat.lst
+++ b/xen/include/xlat.lst
@@ -81,6 +81,7 @@
 !	processor_power			platform.h
 ?	processor_px			platform.h
 !	psd_package			platform.h
+!	remove_from_physmap		memory.h
 ?	xenpf_pcpuinfo			platform.h
 ?	xenpf_pcpu_version		platform.h
 !	sched_poll			sched.h
diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
index df6cec2..566c808 100644
--- a/xen/include/xsm/xsm.h
+++ b/xen/include/xsm/xsm.h
@@ -169,6 +169,7 @@ struct xsm_operations {
     int (*update_va_mapping) (struct domain *d, struct domain *f, 
                                                             l1_pgentry_t pte);
     int (*add_to_physmap) (struct domain *d1, struct domain *d2);
+    int (*remove_from_physmap) (struct domain *d1, struct domain *d2);
     int (*sendtrigger) (struct domain *d);
     int (*bind_pt_irq) (struct domain *d, struct xen_domctl_bind_pt_irq *bind);
     int (*unbind_pt_irq) (struct domain *d);
@@ -738,6 +739,11 @@ static inline int xsm_add_to_physmap(struct domain *d1, struct domain *d2)
     return xsm_call(add_to_physmap(d1, d2));
 }
 
+static inline int xsm_remove_from_physmap(struct domain *d1, struct domain *d2)
+{
+    return xsm_call(remove_from_physmap(d1, d2));
+}
+
 static inline int xsm_sendtrigger(struct domain *d)
 {
     return xsm_call(sendtrigger(d));
diff --git a/xen/xsm/dummy.c b/xen/xsm/dummy.c
index 4bbfbff..65daa4e 100644
--- a/xen/xsm/dummy.c
+++ b/xen/xsm/dummy.c
@@ -529,6 +529,11 @@ static int dummy_add_to_physmap (struct domain *d1, struct domain *d2)
     return 0;
 }
 
+static int dummy_remove_from_physmap (struct domain *d1, struct domain *d2)
+{
+    return 0;
+}
+
 static int dummy_sendtrigger (struct domain *d)
 {
     return 0;
@@ -690,6 +695,7 @@ void xsm_fixup_ops (struct xsm_operations *ops)
     set_to_dummy_if_null(ops, mmu_machphys_update);
     set_to_dummy_if_null(ops, update_va_mapping);
     set_to_dummy_if_null(ops, add_to_physmap);
+    set_to_dummy_if_null(ops, remove_from_physmap);
     set_to_dummy_if_null(ops, sendtrigger);
     set_to_dummy_if_null(ops, bind_pt_irq);
     set_to_dummy_if_null(ops, pin_mem_cacheattr);
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index 0d35767..a2020a9 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -1283,6 +1283,11 @@ static int flask_add_to_physmap(struct domain *d1, struct domain *d2)
     return domain_has_perm(d1, d2, SECCLASS_MMU, MMU__PHYSMAP);
 }
 
+static int flask_remove_from_physmap(struct domain *d1, struct domain *d2)
+{
+    return domain_has_perm(d1, d2, SECCLASS_MMU, MMU__PHYSMAP);
+}
+
 static int flask_sendtrigger(struct domain *d)
 {
     return domain_has_perm(current->domain, d, SECCLASS_DOMAIN, DOMAIN__TRIGGER);
@@ -1550,6 +1555,7 @@ static struct xsm_operations flask_ops = {
     .mmu_machphys_update = flask_mmu_machphys_update,
     .update_va_mapping = flask_update_va_mapping,
     .add_to_physmap = flask_add_to_physmap,
+    .remove_from_physmap = flask_remove_from_physmap,
     .sendtrigger = flask_sendtrigger,
     .get_device_group = flask_get_device_group,
     .test_assign_device = flask_test_assign_device,
-- 
1.7.7.5

Re: [PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Jan Beulich]

>>> On 13.01.12 at 00:35, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> --- a/xen/arch/ia64/xen/mm.c
> +++ b/xen/arch/ia64/xen/mm.c
> @@ -3448,6 +3448,40 @@ arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
>          break;
>      }
>  
> +    case XENMEM_remove_from_physmap:
> +    {
> +        struct xen_remove_from_physmap xrfp;
> +        unsigned long mfn;
> +        struct domain *d;
> +
> +        if ( copy_from_guest(&xrfp, arg, 1) )
> +            return -EFAULT;
> +
> +        rc = rcu_lock_target_domain_by_id(xrfp.domid, &d);
> +        if ( rc != 0 )
> +            return rc;
> +
> +        if ( xsm_remove_from_physmap(current->domain, d) )
> +        {
> +            rcu_unlock_domain(d);
> +            return -EPERM;
> +        }
> +
> +        domain_lock(d);
> +
> +        mfn = gmfn_to_mfn(d, xrfp.gpfn);
> +
> +        if ( mfn_valid(mfn) )
> +            guest_physmap_remove_page(d, xrfp.gpfn, mfn, 0);

Here and in the other similar paths - is it really to be considered
'success' if !mfn_valid(mfn)?

Further, given that the code looks all the same between ia64 and x86
- is this really arch-specific (the get_gfn_untyped() and put_gfn() used
in x86 should really be stubbed out for ia64 anyway)?

> +
> +        domain_unlock(d);
> +
> +        rcu_unlock_domain(d);
> +
> +        break;
> +    }
> +
> +
>      case XENMEM_machine_memory_map:
>      {
>          struct xen_memory_map memmap;
> --- a/xen/include/xlat.lst
> +++ b/xen/include/xlat.lst
> @@ -81,6 +81,7 @@
>  !	processor_power			platform.h
>  ?	processor_px			platform.h
>  !	psd_package			platform.h
> +!	remove_from_physmap		memory.h

Sorry, but this is not how I meant my comment on v1 to be
interpreted - I thought that by looking at the rest of the file it would
go without saying that the sorting is by header name (alphabetically),
then by type name (again alphabetically). (I realize there are other
cases where sorting is not fully correct, but as I'm striving to adjust
this via secondary changes, I'd like to avoid getting the situation
worse with new additions.)

>  ?	xenpf_pcpuinfo			platform.h
>  ?	xenpf_pcpu_version		platform.h
>  !	sched_poll			sched.h

Jan

Re: [PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> 
> This patch reinstates the XENMEM_remove_from_physmap hypercall
> which was removed in 19041:ee62aaafff46 because it was not used.
> 
> However, is now needed in order to support xenstored stub domains.
> The xenstored stub domain is not priviliged like dom0 and so cannot
> unilaterally map the xenbus page of other guests into it's address
> space.  Therefore, before creating a domU the domain builder needs to
> seed its grant table with a grant ref allowing the xenstored stub
> domain to access the new domU's xenbus page.
> 
> At present domU's do not start with their grant table mapped.
> Instead it gets mapped when the guest requests a grant table from
> the hypervisor.
> 
> In order to seed the grant table, the domain builder first needs to
> map it into dom0 address space.  But the hypercall to do this
> requires a gpfn (guest pfn), which is an mfn for PV guest, but a pfn
> for HVM guests.  Therfore, in order to seed the grant table of an
> HVM guest, dom0 needs to *temporarily* map it into the guest's
> "physical" address space.
> 
> Hence the need to reinstate the XENMEM_remove_from_physmap hypercall.
> 
> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

Acked-by: Ian Campbell <ian.campbell@citrix.com> (modulo Jan's comment
about ordering in xlat.lst)

BTW, since Alex and Diego have subsequently left Citrix you could take
my Acked-by's in this series as Signed-of-by on behalf of Citrix. I've
no idea if that's necessary though, I expect not.

Ian.

> ---
>  xen/arch/ia64/xen/mm.c          |   34 ++++++++++++++++++++++++++++++++++
>  xen/arch/x86/mm.c               |   35 +++++++++++++++++++++++++++++++++++
>  xen/arch/x86/x86_64/compat/mm.c |   14 ++++++++++++++
>  xen/include/public/memory.h     |   16 ++++++++++++++++
>  xen/include/xlat.lst            |    1 +
>  xen/include/xsm/xsm.h           |    6 ++++++
>  xen/xsm/dummy.c                 |    6 ++++++
>  xen/xsm/flask/hooks.c           |    6 ++++++
>  8 files changed, 118 insertions(+), 0 deletions(-)
> 
> diff --git a/xen/arch/ia64/xen/mm.c b/xen/arch/ia64/xen/mm.c
> index 84f6a61..a40f96a 100644
> --- a/xen/arch/ia64/xen/mm.c
> +++ b/xen/arch/ia64/xen/mm.c
> @@ -3448,6 +3448,40 @@ arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
>          break;
>      }
>  
> +    case XENMEM_remove_from_physmap:
> +    {
> +        struct xen_remove_from_physmap xrfp;
> +        unsigned long mfn;
> +        struct domain *d;
> +
> +        if ( copy_from_guest(&xrfp, arg, 1) )
> +            return -EFAULT;
> +
> +        rc = rcu_lock_target_domain_by_id(xrfp.domid, &d);
> +        if ( rc != 0 )
> +            return rc;
> +
> +        if ( xsm_remove_from_physmap(current->domain, d) )
> +        {
> +            rcu_unlock_domain(d);
> +            return -EPERM;
> +        }
> +
> +        domain_lock(d);
> +
> +        mfn = gmfn_to_mfn(d, xrfp.gpfn);
> +
> +        if ( mfn_valid(mfn) )
> +            guest_physmap_remove_page(d, xrfp.gpfn, mfn, 0);
> +
> +        domain_unlock(d);
> +
> +        rcu_unlock_domain(d);
> +
> +        break;
> +    }
> +
> +
>      case XENMEM_machine_memory_map:
>      {
>          struct xen_memory_map memmap;
> diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
> index 1f996e0..39cc3c0 100644
> --- a/xen/arch/x86/mm.c
> +++ b/xen/arch/x86/mm.c
> @@ -4871,6 +4871,41 @@ long arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
>          return rc;
>      }
>  
> +    case XENMEM_remove_from_physmap:
> +    {
> +        struct xen_remove_from_physmap xrfp;
> +        unsigned long mfn;
> +        struct domain *d;
> +
> +        if ( copy_from_guest(&xrfp, arg, 1) )
> +            return -EFAULT;
> +
> +        rc = rcu_lock_target_domain_by_id(xrfp.domid, &d);
> +        if ( rc != 0 )
> +            return rc;
> +
> +        if ( xsm_remove_from_physmap(current->domain, d) )
> +        {
> +            rcu_unlock_domain(d);
> +            return -EPERM;
> +        }
> +
> +        domain_lock(d);
> +
> +        mfn = get_gfn_untyped(d, xrfp.gpfn);
> +
> +        if ( mfn_valid(mfn) )
> +            guest_physmap_remove_page(d, xrfp.gpfn, mfn, PAGE_ORDER_4K);
> +
> +        put_gfn(d, xrfp.gpfn);
> +
> +        domain_unlock(d);
> +
> +        rcu_unlock_domain(d);
> +
> +        break;
> +    }
> +
>      case XENMEM_set_memory_map:
>      {
>          struct xen_foreign_memory_map fmap;
> diff --git a/xen/arch/x86/x86_64/compat/mm.c b/xen/arch/x86/x86_64/compat/mm.c
> index bea94fe..dde5430 100644
> --- a/xen/arch/x86/x86_64/compat/mm.c
> +++ b/xen/arch/x86/x86_64/compat/mm.c
> @@ -82,6 +82,20 @@ int compat_arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
>          break;
>      }
>  
> +    case XENMEM_remove_from_physmap:
> +    {
> +        struct compat_remove_from_physmap cmp;
> +        struct xen_remove_from_physmap *nat = (void *)COMPAT_ARG_XLAT_VIRT_BASE;
> +
> +        if ( copy_from_guest(&cmp, arg, 1) )
> +            return -EFAULT;
> +
> +        XLAT_remove_from_physmap(nat, &cmp);
> +        rc = arch_memory_op(op, guest_handle_from_ptr(nat, void));
> +
> +        break;
> +    }
> +
>      case XENMEM_set_memory_map:
>      {
>          struct compat_foreign_memory_map cmp;
> diff --git a/xen/include/public/memory.h b/xen/include/public/memory.h
> index c5b78a8..308deff 100644
> --- a/xen/include/public/memory.h
> +++ b/xen/include/public/memory.h
> @@ -229,6 +229,22 @@ struct xen_add_to_physmap {
>  typedef struct xen_add_to_physmap xen_add_to_physmap_t;
>  DEFINE_XEN_GUEST_HANDLE(xen_add_to_physmap_t);
>  
> +/*
> + * Unmaps the page appearing at a particular GPFN from the specified guest's
> + * pseudophysical address space.
> + * arg == addr of xen_remove_from_physmap_t.
> + */
> +#define XENMEM_remove_from_physmap      15
> +struct xen_remove_from_physmap {
> +    /* Which domain to change the mapping for. */
> +    domid_t domid;
> +
> +    /* GPFN of the current mapping of the page. */
> +    xen_pfn_t     gpfn;
> +};
> +typedef struct xen_remove_from_physmap xen_remove_from_physmap_t;
> +DEFINE_XEN_GUEST_HANDLE(xen_remove_from_physmap_t);
> +
>  /*** REMOVED ***/
>  /*#define XENMEM_translate_gpfn_list  8*/
>  
> diff --git a/xen/include/xlat.lst b/xen/include/xlat.lst
> index 3d92175..ee1772c 100644
> --- a/xen/include/xlat.lst
> +++ b/xen/include/xlat.lst
> @@ -81,6 +81,7 @@
>  !	processor_power			platform.h
>  ?	processor_px			platform.h
>  !	psd_package			platform.h
> +!	remove_from_physmap		memory.h
>  ?	xenpf_pcpuinfo			platform.h
>  ?	xenpf_pcpu_version		platform.h
>  !	sched_poll			sched.h
> diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
> index df6cec2..566c808 100644
> --- a/xen/include/xsm/xsm.h
> +++ b/xen/include/xsm/xsm.h
> @@ -169,6 +169,7 @@ struct xsm_operations {
>      int (*update_va_mapping) (struct domain *d, struct domain *f, 
>                                                              l1_pgentry_t pte);
>      int (*add_to_physmap) (struct domain *d1, struct domain *d2);
> +    int (*remove_from_physmap) (struct domain *d1, struct domain *d2);
>      int (*sendtrigger) (struct domain *d);
>      int (*bind_pt_irq) (struct domain *d, struct xen_domctl_bind_pt_irq *bind);
>      int (*unbind_pt_irq) (struct domain *d);
> @@ -738,6 +739,11 @@ static inline int xsm_add_to_physmap(struct domain *d1, struct domain *d2)
>      return xsm_call(add_to_physmap(d1, d2));
>  }
>  
> +static inline int xsm_remove_from_physmap(struct domain *d1, struct domain *d2)
> +{
> +    return xsm_call(remove_from_physmap(d1, d2));
> +}
> +
>  static inline int xsm_sendtrigger(struct domain *d)
>  {
>      return xsm_call(sendtrigger(d));
> diff --git a/xen/xsm/dummy.c b/xen/xsm/dummy.c
> index 4bbfbff..65daa4e 100644
> --- a/xen/xsm/dummy.c
> +++ b/xen/xsm/dummy.c
> @@ -529,6 +529,11 @@ static int dummy_add_to_physmap (struct domain *d1, struct domain *d2)
>      return 0;
>  }
>  
> +static int dummy_remove_from_physmap (struct domain *d1, struct domain *d2)
> +{
> +    return 0;
> +}
> +
>  static int dummy_sendtrigger (struct domain *d)
>  {
>      return 0;
> @@ -690,6 +695,7 @@ void xsm_fixup_ops (struct xsm_operations *ops)
>      set_to_dummy_if_null(ops, mmu_machphys_update);
>      set_to_dummy_if_null(ops, update_va_mapping);
>      set_to_dummy_if_null(ops, add_to_physmap);
> +    set_to_dummy_if_null(ops, remove_from_physmap);
>      set_to_dummy_if_null(ops, sendtrigger);
>      set_to_dummy_if_null(ops, bind_pt_irq);
>      set_to_dummy_if_null(ops, pin_mem_cacheattr);
> diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
> index 0d35767..a2020a9 100644
> --- a/xen/xsm/flask/hooks.c
> +++ b/xen/xsm/flask/hooks.c
> @@ -1283,6 +1283,11 @@ static int flask_add_to_physmap(struct domain *d1, struct domain *d2)
>      return domain_has_perm(d1, d2, SECCLASS_MMU, MMU__PHYSMAP);
>  }
>  
> +static int flask_remove_from_physmap(struct domain *d1, struct domain *d2)
> +{
> +    return domain_has_perm(d1, d2, SECCLASS_MMU, MMU__PHYSMAP);
> +}
> +
>  static int flask_sendtrigger(struct domain *d)
>  {
>      return domain_has_perm(current->domain, d, SECCLASS_DOMAIN, DOMAIN__TRIGGER);
> @@ -1550,6 +1555,7 @@ static struct xsm_operations flask_ops = {
>      .mmu_machphys_update = flask_mmu_machphys_update,
>      .update_va_mapping = flask_update_va_mapping,
>      .add_to_physmap = flask_add_to_physmap,
> +    .remove_from_physmap = flask_remove_from_physmap,
>      .sendtrigger = flask_sendtrigger,
>      .get_device_group = flask_get_device_group,
>      .test_assign_device = flask_test_assign_device,

Re: [PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Daniel De Graaf]

On 01/18/2012 05:36 AM, Ian Campbell wrote:
> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
>> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
>>
>> This patch reinstates the XENMEM_remove_from_physmap hypercall
>> which was removed in 19041:ee62aaafff46 because it was not used.
>>
>> However, is now needed in order to support xenstored stub domains.
>> The xenstored stub domain is not priviliged like dom0 and so cannot
>> unilaterally map the xenbus page of other guests into it's address
>> space.  Therefore, before creating a domU the domain builder needs to
>> seed its grant table with a grant ref allowing the xenstored stub
>> domain to access the new domU's xenbus page.
>>
>> At present domU's do not start with their grant table mapped.
>> Instead it gets mapped when the guest requests a grant table from
>> the hypervisor.
>>
>> In order to seed the grant table, the domain builder first needs to
>> map it into dom0 address space.  But the hypercall to do this
>> requires a gpfn (guest pfn), which is an mfn for PV guest, but a pfn
>> for HVM guests.  Therfore, in order to seed the grant table of an
>> HVM guest, dom0 needs to *temporarily* map it into the guest's
>> "physical" address space.
>>
>> Hence the need to reinstate the XENMEM_remove_from_physmap hypercall.
>>
>> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> 
> Acked-by: Ian Campbell <ian.campbell@citrix.com> (modulo Jan's comment
> about ordering in xlat.lst)
> 
> BTW, since Alex and Diego have subsequently left Citrix you could take
> my Acked-by's in this series as Signed-of-by on behalf of Citrix. I've
> no idea if that's necessary though, I expect not.
> 
> Ian.
> 

I'm not an expert in this area, but this is how I read it: the portion of
the path authored by Alex/Diego was already signed-off when they were posted,
so since the current patches are derived works from them the sign-off may
need to stay in order to allow me to sign off because I cannot claim copyright
on all of the content. Assuming Citrix actually owns the copyright on the
patches, your Ack may suffice to replace the sign-off for this purpose.

I guess my real question here would be: should the sign-off from Alex and
Diego remain on these patches in addition to your Ack?

Re: [PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Ian Campbell]

On Wed, 2012-01-18 at 14:56 +0000, Daniel De Graaf wrote:
> On 01/18/2012 05:36 AM, Ian Campbell wrote:
> > On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> >> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> >>
> >> This patch reinstates the XENMEM_remove_from_physmap hypercall
> >> which was removed in 19041:ee62aaafff46 because it was not used.
> >>
> >> However, is now needed in order to support xenstored stub domains.
> >> The xenstored stub domain is not priviliged like dom0 and so cannot
> >> unilaterally map the xenbus page of other guests into it's address
> >> space.  Therefore, before creating a domU the domain builder needs to
> >> seed its grant table with a grant ref allowing the xenstored stub
> >> domain to access the new domU's xenbus page.
> >>
> >> At present domU's do not start with their grant table mapped.
> >> Instead it gets mapped when the guest requests a grant table from
> >> the hypervisor.
> >>
> >> In order to seed the grant table, the domain builder first needs to
> >> map it into dom0 address space.  But the hypercall to do this
> >> requires a gpfn (guest pfn), which is an mfn for PV guest, but a pfn
> >> for HVM guests.  Therfore, in order to seed the grant table of an
> >> HVM guest, dom0 needs to *temporarily* map it into the guest's
> >> "physical" address space.
> >>
> >> Hence the need to reinstate the XENMEM_remove_from_physmap hypercall.
> >>
> >> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> >> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> > 
> > Acked-by: Ian Campbell <ian.campbell@citrix.com> (modulo Jan's comment
> > about ordering in xlat.lst)
> > 
> > BTW, since Alex and Diego have subsequently left Citrix you could take
> > my Acked-by's in this series as Signed-of-by on behalf of Citrix. I've
> > no idea if that's necessary though, I expect not.
> > 
> > Ian.
> > 
> 
> I'm not an expert in this area,

Me neither.

> but this is how I read it: the portion of
> the path authored by Alex/Diego was already signed-off when they were posted,
> so since the current patches are derived works from them the sign-off may
> need to stay in order to allow me to sign off because I cannot claim copyright
> on all of the content. Assuming Citrix actually owns the copyright on the
> patches, your Ack may suffice to replace the sign-off for this purpose.

I don't think an Ack conveys the same meaning (WRT the DCO) as a
Signed-off-by.

> I guess my real question here would be: should the sign-off from Alex and
> Diego remain on these patches in addition to your Ack?

I would suggest you keep any signed-off-by they provided and augment it
with my ack. 

I think I saw one or two which said "Originally-by" instead of
"Signed-of-by", I guess those were either missing a Signed-off-by in the
first place or have been heavily modified?

Ian.

> 

Re: [PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Daniel De Graaf]

On 01/18/2012 11:06 AM, Ian Campbell wrote:
> On Wed, 2012-01-18 at 14:56 +0000, Daniel De Graaf wrote:
>> On 01/18/2012 05:36 AM, Ian Campbell wrote:
>>> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
>>>> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
>>>>
>>>> This patch reinstates the XENMEM_remove_from_physmap hypercall
>>>> which was removed in 19041:ee62aaafff46 because it was not used.
>>>>
>>>> However, is now needed in order to support xenstored stub domains.
>>>> The xenstored stub domain is not priviliged like dom0 and so cannot
>>>> unilaterally map the xenbus page of other guests into it's address
>>>> space.  Therefore, before creating a domU the domain builder needs to
>>>> seed its grant table with a grant ref allowing the xenstored stub
>>>> domain to access the new domU's xenbus page.
>>>>
>>>> At present domU's do not start with their grant table mapped.
>>>> Instead it gets mapped when the guest requests a grant table from
>>>> the hypervisor.
>>>>
>>>> In order to seed the grant table, the domain builder first needs to
>>>> map it into dom0 address space.  But the hypercall to do this
>>>> requires a gpfn (guest pfn), which is an mfn for PV guest, but a pfn
>>>> for HVM guests.  Therfore, in order to seed the grant table of an
>>>> HVM guest, dom0 needs to *temporarily* map it into the guest's
>>>> "physical" address space.
>>>>
>>>> Hence the need to reinstate the XENMEM_remove_from_physmap hypercall.
>>>>
>>>> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
>>>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>>>
>>> Acked-by: Ian Campbell <ian.campbell@citrix.com> (modulo Jan's comment
>>> about ordering in xlat.lst)
>>>
>>> BTW, since Alex and Diego have subsequently left Citrix you could take
>>> my Acked-by's in this series as Signed-of-by on behalf of Citrix. I've
>>> no idea if that's necessary though, I expect not.
>>>
>>> Ian.
>>>
>>
>> I'm not an expert in this area,
> 
> Me neither.
> 
>> but this is how I read it: the portion of
>> the path authored by Alex/Diego was already signed-off when they were posted,
>> so since the current patches are derived works from them the sign-off may
>> need to stay in order to allow me to sign off because I cannot claim copyright
>> on all of the content. Assuming Citrix actually owns the copyright on the
>> patches, your Ack may suffice to replace the sign-off for this purpose.
> 
> I don't think an Ack conveys the same meaning (WRT the DCO) as a
> Signed-off-by.
> 
>> I guess my real question here would be: should the sign-off from Alex and
>> Diego remain on these patches in addition to your Ack?
> 
> I would suggest you keep any signed-off-by they provided and augment it
> with my ack. 
> 
> I think I saw one or two which said "Originally-by" instead of
> "Signed-of-by", I guess those were either missing a Signed-off-by in the
> first place or have been heavily modified?
> 
> Ian.
> 

I originally replaced all the signed-off-by lines with originally-by and
missed one when converting back. When looking at the Linux version of the
DCO, it implies (lower down when talking about subsystem maintainers) that
if I make changes I need to drop the sign-off and claim clause (b) unless
the original author is around to sign-off on the changed patch, or if it is
trivial and I note this above my sign-off (not applicable here). This makes
me lean toward changing back to "Originally-by" or similar tags. I did keep
the From tags for those patches that I did not mostly rewrite, which I assume
will be recognized when importing patches.

-- 
Daniel De Graaf
National Security Agency

Re: [PATCH 01/18] xen: reinstate previously unused XENMEM_remove_from_physmap hypercall

[Ian Campbell]

On Wed, 2012-01-18 at 19:07 +0000, Daniel De Graaf wrote:
> On 01/18/2012 11:06 AM, Ian Campbell wrote:
> > On Wed, 2012-01-18 at 14:56 +0000, Daniel De Graaf wrote:
> >> On 01/18/2012 05:36 AM, Ian Campbell wrote:
> >>> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> >>>> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> >>>>
> >>>> This patch reinstates the XENMEM_remove_from_physmap hypercall
> >>>> which was removed in 19041:ee62aaafff46 because it was not used.
> >>>>
> >>>> However, is now needed in order to support xenstored stub domains.
> >>>> The xenstored stub domain is not priviliged like dom0 and so cannot
> >>>> unilaterally map the xenbus page of other guests into it's address
> >>>> space.  Therefore, before creating a domU the domain builder needs to
> >>>> seed its grant table with a grant ref allowing the xenstored stub
> >>>> domain to access the new domU's xenbus page.
> >>>>
> >>>> At present domU's do not start with their grant table mapped.
> >>>> Instead it gets mapped when the guest requests a grant table from
> >>>> the hypervisor.
> >>>>
> >>>> In order to seed the grant table, the domain builder first needs to
> >>>> map it into dom0 address space.  But the hypercall to do this
> >>>> requires a gpfn (guest pfn), which is an mfn for PV guest, but a pfn
> >>>> for HVM guests.  Therfore, in order to seed the grant table of an
> >>>> HVM guest, dom0 needs to *temporarily* map it into the guest's
> >>>> "physical" address space.
> >>>>
> >>>> Hence the need to reinstate the XENMEM_remove_from_physmap hypercall.
> >>>>
> >>>> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> >>>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> >>>
> >>> Acked-by: Ian Campbell <ian.campbell@citrix.com> (modulo Jan's comment
> >>> about ordering in xlat.lst)
> >>>
> >>> BTW, since Alex and Diego have subsequently left Citrix you could take
> >>> my Acked-by's in this series as Signed-of-by on behalf of Citrix. I've
> >>> no idea if that's necessary though, I expect not.
> >>>
> >>> Ian.
> >>>
> >>
> >> I'm not an expert in this area,
> > 
> > Me neither.
> > 
> >> but this is how I read it: the portion of
> >> the path authored by Alex/Diego was already signed-off when they were posted,
> >> so since the current patches are derived works from them the sign-off may
> >> need to stay in order to allow me to sign off because I cannot claim copyright
> >> on all of the content. Assuming Citrix actually owns the copyright on the
> >> patches, your Ack may suffice to replace the sign-off for this purpose.
> > 
> > I don't think an Ack conveys the same meaning (WRT the DCO) as a
> > Signed-off-by.
> > 
> >> I guess my real question here would be: should the sign-off from Alex and
> >> Diego remain on these patches in addition to your Ack?
> > 
> > I would suggest you keep any signed-off-by they provided and augment it
> > with my ack. 
> > 
> > I think I saw one or two which said "Originally-by" instead of
> > "Signed-of-by", I guess those were either missing a Signed-off-by in the
> > first place or have been heavily modified?
> > 
> > Ian.
> > 
> 
> I originally replaced all the signed-off-by lines with originally-by and
> missed one when converting back. When looking at the Linux version of the
> DCO, it implies (lower down when talking about subsystem maintainers) that
> if I make changes I need to drop the sign-off and claim clause (b) unless
> the original author is around to sign-off on the changed patch, or if it is
> trivial and I note this above my sign-off (not applicable here). This makes
> me lean toward changing back to "Originally-by" or similar tags. I did keep
> the From tags for those patches that I did not mostly rewrite, which I assume
> will be recognized when importing patches.

The DCO itself isn't terribly specific about what to do with an existing
Signed-off-by if you modify the patch. Common practice appears to be to
include both the original and your own and to note what you have changed
unless you have done a wholesale rewrite in which case it is
"Based-on"/"Originally-by"/etc + your own S-o-b.

Ian.


> 

[PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Daniel De Graaf]

This patch sends global VIRQs to a domain designated as the VIRQ handler
instead of sending all global VIRQ events to dom0. This is required in
order to run xenstored in a stubdom, because VIRQ_DOM_EXC must be sent
to xenstored for domain destruction to work properly.

This patch was inspired by the xenstored stubdomain patch series sent to
xen-devel by Alex Zeffertt in 2009.

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/flask/policy/policy/flask/access_vectors |    1 +
 tools/libxc/xc_domain.c                        |   10 ++++
 tools/libxc/xenctrl.h                          |    9 +++
 xen/arch/x86/cpu/mcheck/amd_nonfatal.c         |    2 +-
 xen/arch/x86/cpu/mcheck/mce.c                  |    2 +-
 xen/arch/x86/cpu/mcheck/mce_intel.c            |    6 +-
 xen/arch/x86/cpu/mcheck/non-fatal.c            |    2 +-
 xen/common/cpu.c                               |    4 +-
 xen/common/domain.c                            |    8 ++--
 xen/common/domctl.c                            |   17 ++++++
 xen/common/event_channel.c                     |   63 +++++++++++++++++++++++-
 xen/common/trace.c                             |    2 +-
 xen/drivers/char/console.c                     |    4 +-
 xen/include/public/domctl.h                    |    8 +++
 xen/include/xen/event.h                        |   12 +++-
 xen/include/xsm/xsm.h                          |    6 ++
 xen/xsm/dummy.c                                |    6 ++
 xen/xsm/flask/hooks.c                          |    6 ++
 xen/xsm/flask/include/av_perm_to_string.h      |    1 +
 xen/xsm/flask/include/av_permissions.h         |    1 +
 20 files changed, 151 insertions(+), 19 deletions(-)

diff --git a/tools/flask/policy/policy/flask/access_vectors b/tools/flask/policy/policy/flask/access_vectors
index 644f2e1..5901911 100644
--- a/tools/flask/policy/policy/flask/access_vectors
+++ b/tools/flask/policy/policy/flask/access_vectors
@@ -85,6 +85,7 @@ class domain
 	getpodtarget
 	setpodtarget
 	set_misc_info
+	set_virq_handler
 }
 
 class hvm
diff --git a/tools/libxc/xc_domain.c b/tools/libxc/xc_domain.c
index ab019b8..d98e68b 100644
--- a/tools/libxc/xc_domain.c
+++ b/tools/libxc/xc_domain.c
@@ -1504,6 +1504,16 @@ int xc_domain_set_access_required(xc_interface *xch,
     return do_domctl(xch, &domctl);
 }
 
+int xc_domain_set_virq_handler(xc_interface *xch, uint32_t domid, int virq)
+{
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_set_virq_handler;
+    domctl.domain = domid;
+    domctl.u.set_virq_handler.virq = virq;
+    return do_domctl(xch, &domctl);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxc/xenctrl.h b/tools/libxc/xenctrl.h
index 8b34769..8f3426f 100644
--- a/tools/libxc/xenctrl.h
+++ b/tools/libxc/xenctrl.h
@@ -747,6 +747,15 @@ int xc_domain_p2m_audit(xc_interface *xch,
 int xc_domain_set_access_required(xc_interface *xch,
 				  uint32_t domid,
 				  unsigned int required);
+/**
+ * This function sets the handler of global VIRQs sent by the hypervisor
+ *
+ * @parm xch a handle to an open hypervisor interface
+ * @parm domid the domain id which will handle the VIRQ
+ * @parm virq the virq number (VIRQ_*)
+ * return 0 on success, -1 on failure
+ */
+int xc_domain_set_virq_handler(xc_interface *xch, uint32_t domid, int virq);
 
 /*
  * CPUPOOL MANAGEMENT FUNCTIONS
diff --git a/xen/arch/x86/cpu/mcheck/amd_nonfatal.c b/xen/arch/x86/cpu/mcheck/amd_nonfatal.c
index 50288bd..9222098 100644
--- a/xen/arch/x86/cpu/mcheck/amd_nonfatal.c
+++ b/xen/arch/x86/cpu/mcheck/amd_nonfatal.c
@@ -100,7 +100,7 @@ static void mce_amd_checkregs(void *info)
 
 		if (dom0_vmce_enabled()) {
 			mctelem_commit(mctc);
-			send_guest_global_virq(dom0, VIRQ_MCA);
+			send_global_virq(VIRQ_MCA);
 		} else if (++dumpcount >= 10) {
 			x86_mcinfo_dump((struct mc_info *)mctelem_dataptr(mctc));
 			mctelem_dismiss(mctc);
diff --git a/xen/arch/x86/cpu/mcheck/mce.c b/xen/arch/x86/cpu/mcheck/mce.c
index b592041..c4e4477 100644
--- a/xen/arch/x86/cpu/mcheck/mce.c
+++ b/xen/arch/x86/cpu/mcheck/mce.c
@@ -594,7 +594,7 @@ void mcheck_cmn_handler(struct cpu_user_regs *regs, long error_code,
         if (dom0_vmce_enabled()) {
             if (mctc != NULL)
                 mctelem_commit(mctc);
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         } else {
             x86_mcinfo_dump(mci);
             if (mctc != NULL)
diff --git a/xen/arch/x86/cpu/mcheck/mce_intel.c b/xen/arch/x86/cpu/mcheck/mce_intel.c
index 0986025..0894080 100644
--- a/xen/arch/x86/cpu/mcheck/mce_intel.c
+++ b/xen/arch/x86/cpu/mcheck/mce_intel.c
@@ -354,7 +354,7 @@ static void mce_softirq(void)
         /* Step2: Send Log to DOM0 through vIRQ */
         if (dom0_vmce_enabled()) {
             mce_printk(MCE_VERBOSE, "MCE: send MCE# to DOM0 through virq\n");
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         }
     }
 
@@ -1085,7 +1085,7 @@ static void cmci_discover(void)
     if (bs.errcnt && mctc != NULL) {
         if (dom0_vmce_enabled()) {
             mctelem_commit(mctc);
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         } else {
             x86_mcinfo_dump(mctelem_dataptr(mctc));
             mctelem_dismiss(mctc);
@@ -1205,7 +1205,7 @@ fastcall void smp_cmci_interrupt(struct cpu_user_regs *regs)
         if (dom0_vmce_enabled()) {
             mctelem_commit(mctc);
             mce_printk(MCE_VERBOSE, "CMCI: send CMCI to DOM0 through virq\n");
-            send_guest_global_virq(dom0, VIRQ_MCA);
+            send_global_virq(VIRQ_MCA);
         } else {
             x86_mcinfo_dump(mctelem_dataptr(mctc));
             mctelem_dismiss(mctc);
diff --git a/xen/arch/x86/cpu/mcheck/non-fatal.c b/xen/arch/x86/cpu/mcheck/non-fatal.c
index c57688f..1dded9b 100644
--- a/xen/arch/x86/cpu/mcheck/non-fatal.c
+++ b/xen/arch/x86/cpu/mcheck/non-fatal.c
@@ -55,7 +55,7 @@ static void mce_checkregs (void *info)
 
 		if (dom0_vmce_enabled()) {
 			mctelem_commit(mctc);
-			send_guest_global_virq(dom0, VIRQ_MCA);
+			send_global_virq(VIRQ_MCA);
 		} else if (++dumpcount >= 10) {
 			x86_mcinfo_dump((struct mc_info *)mctelem_dataptr(mctc));
 			mctelem_dismiss(mctc);
diff --git a/xen/common/cpu.c b/xen/common/cpu.c
index 79abdb7..630881e 100644
--- a/xen/common/cpu.c
+++ b/xen/common/cpu.c
@@ -108,7 +108,7 @@ int cpu_down(unsigned int cpu)
     notifier_rc = notifier_call_chain(&cpu_chain, CPU_DEAD, hcpu, NULL);
     BUG_ON(notifier_rc != NOTIFY_DONE);
 
-    send_guest_global_virq(dom0, VIRQ_PCPU_STATE);
+    send_global_virq(VIRQ_PCPU_STATE);
     cpu_hotplug_done();
     return 0;
 
@@ -148,7 +148,7 @@ int cpu_up(unsigned int cpu)
     notifier_rc = notifier_call_chain(&cpu_chain, CPU_ONLINE, hcpu, NULL);
     BUG_ON(notifier_rc != NOTIFY_DONE);
 
-    send_guest_global_virq(dom0, VIRQ_PCPU_STATE);
+    send_global_virq(VIRQ_PCPU_STATE);
 
     cpu_hotplug_done();
     return 0;
diff --git a/xen/common/domain.c b/xen/common/domain.c
index 52a63ef..f1a7ede 100644
--- a/xen/common/domain.c
+++ b/xen/common/domain.c
@@ -116,7 +116,7 @@ static void __domain_finalise_shutdown(struct domain *d)
     if ( (d->shutdown_code == SHUTDOWN_suspend) && d->suspend_evtchn )
         evtchn_send(d, d->suspend_evtchn);
     else
-        send_guest_global_virq(dom0, VIRQ_DOM_EXC);
+        send_global_virq(VIRQ_DOM_EXC);
 }
 
 static void vcpu_check_shutdown(struct vcpu *v)
@@ -492,7 +492,7 @@ int domain_kill(struct domain *d)
         }
         d->is_dying = DOMDYING_dead;
         put_domain(d);
-        send_guest_global_virq(dom0, VIRQ_DOM_EXC);
+        send_global_virq(VIRQ_DOM_EXC);
         /* fallthrough */
     case DOMDYING_dead:
         break;
@@ -633,7 +633,7 @@ void domain_pause_for_debugger(void)
     for_each_vcpu ( d, v )
         vcpu_sleep_nosync(v);
 
-    send_guest_global_virq(dom0, VIRQ_DEBUGGER);
+    send_global_virq(VIRQ_DEBUGGER);
 }
 
 /* Complete domain destroy after RCU readers are not holding old references. */
@@ -690,7 +690,7 @@ static void complete_domain_destroy(struct rcu_head *head)
     free_cpumask_var(d->domain_dirty_cpumask);
     free_domain_struct(d);
 
-    send_guest_global_virq(dom0, VIRQ_DOM_EXC);
+    send_global_virq(VIRQ_DOM_EXC);
 }
 
 /* Release resources belonging to task @p. */
diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index d6ae09b..97c7d53 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -994,6 +994,23 @@ long do_domctl(XEN_GUEST_HANDLE(xen_domctl_t) u_domctl)
     }
     break;
 
+    case XEN_DOMCTL_set_virq_handler:
+    {
+        struct domain *d;
+        uint32_t virq = op->u.set_virq_handler.virq;
+
+        ret = -ESRCH;
+        d = rcu_lock_domain_by_id(op->domain);
+        if ( d != NULL )
+        {
+            ret = xsm_set_virq_handler(d, virq);
+            if ( !ret )
+                ret = set_global_virq_handler(d, virq);
+            rcu_unlock_domain(d);
+        }
+    }
+    break;
+
     default:
         ret = arch_do_domctl(op, u_domctl);
         break;
diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c
index 9212042..e507481 100644
--- a/xen/common/event_channel.c
+++ b/xen/common/event_channel.c
@@ -689,7 +689,7 @@ void send_guest_vcpu_virq(struct vcpu *v, int virq)
     spin_unlock_irqrestore(&v->virq_lock, flags);
 }
 
-void send_guest_global_virq(struct domain *d, int virq)
+static void send_guest_global_virq(struct domain *d, int virq)
 {
     unsigned long flags;
     int port;
@@ -739,6 +739,65 @@ int send_guest_pirq(struct domain *d, const struct pirq *pirq)
     return evtchn_set_pending(d->vcpu[chn->notify_vcpu_id], port);
 }
 
+static struct domain *global_virq_handlers[NR_VIRQS] __read_mostly;
+
+static DEFINE_SPINLOCK(global_virq_handlers_lock);
+
+void send_global_virq(uint32_t virq)
+{
+    ASSERT(virq < NR_VIRQS);
+    ASSERT(virq_is_global(virq));
+
+    send_guest_global_virq(global_virq_handlers[virq] ?: dom0, virq);
+}
+
+int set_global_virq_handler(struct domain *d, uint32_t virq)
+{
+    struct domain *old;
+
+    if (virq >= NR_VIRQS)
+        return -EINVAL;
+    if (!virq_is_global(virq))
+        return -EINVAL;
+
+    if (global_virq_handlers[virq] == d)
+        return 0;
+
+    if (unlikely(!get_domain(d)))
+        return -EINVAL;
+
+    spin_lock(&global_virq_handlers_lock);
+    old = global_virq_handlers[virq];
+    global_virq_handlers[virq] = d;
+    spin_unlock(&global_virq_handlers_lock);
+
+    if (old != NULL)
+        put_domain(old);
+
+    return 0;
+}
+
+static void clear_global_virq_handlers(struct domain *d)
+{
+    uint32_t virq;
+    int put_count = 0;
+
+    spin_lock(&global_virq_handlers_lock);
+
+    for (virq = 0; virq < NR_VIRQS; virq++) {
+        if (global_virq_handlers[virq] == d) {
+            global_virq_handlers[virq] = NULL;
+            put_count++;
+        }
+    }
+
+    spin_unlock(&global_virq_handlers_lock);
+
+    while (put_count) {
+        put_domain(d);
+        put_count--;
+    }
+}
 
 static long evtchn_status(evtchn_status_t *status)
 {
@@ -1160,6 +1219,8 @@ void evtchn_destroy(struct domain *d)
         d->evtchn[i] = NULL;
     }
     spin_unlock(&d->event_lock);
+
+    clear_global_virq_handlers(d);
 }
 
 
diff --git a/xen/common/trace.c b/xen/common/trace.c
index 5772f24..58cbf39 100644
--- a/xen/common/trace.c
+++ b/xen/common/trace.c
@@ -661,7 +661,7 @@ static inline void insert_lost_records(struct t_buf *buf)
  */
 static void trace_notify_dom0(unsigned long unused)
 {
-    send_guest_global_virq(dom0, VIRQ_TBUF);
+    send_global_virq(VIRQ_TBUF);
 }
 static DECLARE_SOFTIRQ_TASKLET(trace_notify_dom0_tasklet,
                                trace_notify_dom0, 0);
diff --git a/xen/drivers/char/console.c b/xen/drivers/char/console.c
index 8a4c684..79b266f 100644
--- a/xen/drivers/char/console.c
+++ b/xen/drivers/char/console.c
@@ -287,7 +287,7 @@ static void __serial_rx(char c, struct cpu_user_regs *regs)
     if ( (serial_rx_prod-serial_rx_cons) != SERIAL_RX_SIZE )
         serial_rx_ring[SERIAL_RX_MASK(serial_rx_prod++)] = c;
     /* Always notify the guest: prevents receive path from getting stuck. */
-    send_guest_global_virq(dom0, VIRQ_CONSOLE);
+    send_global_virq(VIRQ_CONSOLE);
 }
 
 static void serial_rx(char c, struct cpu_user_regs *regs)
@@ -314,7 +314,7 @@ static void serial_rx(char c, struct cpu_user_regs *regs)
 
 static void notify_dom0_con_ring(unsigned long unused)
 {
-    send_guest_global_virq(dom0, VIRQ_CON_RING);
+    send_global_virq(VIRQ_CON_RING);
 }
 static DECLARE_SOFTIRQ_TASKLET(notify_dom0_con_ring_tasklet,
                                notify_dom0_con_ring, 0);
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index c7640aa..75be370 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -813,6 +813,12 @@ struct xen_domctl_audit_p2m {
 typedef struct xen_domctl_audit_p2m xen_domctl_audit_p2m_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_audit_p2m_t);
 
+struct xen_domctl_set_virq_handler {
+    uint32_t virq; /* IN */
+};
+typedef struct xen_domctl_set_virq_handler xen_domctl_set_virq_handler_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_virq_handler_t);
+
 #if defined(__i386__) || defined(__x86_64__)
 /* XEN_DOMCTL_setvcpuextstate */
 /* XEN_DOMCTL_getvcpuextstate */
@@ -912,6 +918,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_getvcpuextstate               63
 #define XEN_DOMCTL_set_access_required           64
 #define XEN_DOMCTL_audit_p2m                     65
+#define XEN_DOMCTL_set_virq_handler              66
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002
@@ -966,6 +973,7 @@ struct xen_domctl {
 #endif
         struct xen_domctl_set_access_required access_required;
         struct xen_domctl_audit_p2m         audit_p2m;
+        struct xen_domctl_set_virq_handler  set_virq_handler;
         struct xen_domctl_gdbsx_memio       gdbsx_guest_memio;
         struct xen_domctl_gdbsx_pauseunp_vcpu gdbsx_pauseunp_vcpu;
         struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
diff --git a/xen/include/xen/event.h b/xen/include/xen/event.h
index 7e5ad7b..d505ee9 100644
--- a/xen/include/xen/event.h
+++ b/xen/include/xen/event.h
@@ -24,11 +24,17 @@
 void send_guest_vcpu_virq(struct vcpu *v, int virq);
 
 /*
- * send_guest_global_virq: Notify guest via a global VIRQ.
- *  @d:        Domain to which virtual IRQ should be sent
+ * send_global_virq: Notify the domain handling a global VIRQ.
  *  @virq:     Virtual IRQ number (VIRQ_*)
  */
-void send_guest_global_virq(struct domain *d, int virq);
+void send_global_virq(uint32_t virq);
+
+/*
+ * sent_global_virq_handler: Set a global VIRQ handler.
+ *  @d:        New target domain for this VIRQ
+ *  @virq:     Virtual IRQ number (VIRQ_*), must be global
+ */
+int set_global_virq_handler(struct domain *d, uint32_t virq);
 
 /*
  * send_guest_pirq:
diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h
index 566c808..c89c6ed 100644
--- a/xen/include/xsm/xsm.h
+++ b/xen/include/xsm/xsm.h
@@ -64,6 +64,7 @@ struct xsm_operations {
     int (*domain_settime) (struct domain *d);
     int (*set_target) (struct domain *d, struct domain *e);
     int (*domctl) (struct domain *d, int cmd);
+    int (*set_virq_handler) (struct domain *d, int virq);
     int (*tbufcontrol) (void);
     int (*readconsole) (uint32_t clear);
     int (*sched_id) (void);
@@ -265,6 +266,11 @@ static inline int xsm_domctl (struct domain *d, int cmd)
     return xsm_call(domctl(d, cmd));
 }
 
+static inline int xsm_set_virq_handler (struct domain *d, int virq)
+{
+    return xsm_call(set_virq_handler(d, virq));
+}
+
 static inline int xsm_tbufcontrol (void)
 {
     return xsm_call(tbufcontrol());
diff --git a/xen/xsm/dummy.c b/xen/xsm/dummy.c
index 65daa4e..59db86d 100644
--- a/xen/xsm/dummy.c
+++ b/xen/xsm/dummy.c
@@ -94,6 +94,11 @@ static int dummy_domctl(struct domain *d, int cmd)
     return 0;
 }
 
+static int dummy_set_virq_handler(struct domain *d, int virq)
+{
+    return 0;
+}
+
 static int dummy_tbufcontrol (void)
 {
     return 0;
@@ -596,6 +601,7 @@ void xsm_fixup_ops (struct xsm_operations *ops)
     set_to_dummy_if_null(ops, domain_settime);
     set_to_dummy_if_null(ops, set_target);
     set_to_dummy_if_null(ops, domctl);
+    set_to_dummy_if_null(ops, set_virq_handler);
     set_to_dummy_if_null(ops, tbufcontrol);
     set_to_dummy_if_null(ops, readconsole);
     set_to_dummy_if_null(ops, sched_id);
diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c
index a2020a9..a1feb8d 100644
--- a/xen/xsm/flask/hooks.c
+++ b/xen/xsm/flask/hooks.c
@@ -597,6 +597,11 @@ static int flask_domctl(struct domain *d, int cmd)
     return domain_has_perm(current->domain, d, SECCLASS_DOMAIN, DOMAIN__SET_MISC_INFO);
 }
 
+static int flask_set_virq_handler(struct domain *d, int virq)
+{
+    return domain_has_perm(current->domain, d, SECCLASS_DOMAIN, DOMAIN__SET_VIRQ_HANDLER);
+}
+
 static int flask_tbufcontrol(void)
 {
     return domain_has_xen(current->domain, XEN__TBUFCONTROL);
@@ -1460,6 +1465,7 @@ static struct xsm_operations flask_ops = {
     .domain_settime = flask_domain_settime,
     .set_target = flask_set_target,
     .domctl = flask_domctl,
+    .set_virq_handler = flask_set_virq_handler,
     .tbufcontrol = flask_tbufcontrol,
     .readconsole = flask_readconsole,
     .sched_id = flask_sched_id,
diff --git a/xen/xsm/flask/include/av_perm_to_string.h b/xen/xsm/flask/include/av_perm_to_string.h
index 85cbffc..17a1c36 100644
--- a/xen/xsm/flask/include/av_perm_to_string.h
+++ b/xen/xsm/flask/include/av_perm_to_string.h
@@ -60,6 +60,7 @@
    S_(SECCLASS_DOMAIN, DOMAIN__GETPODTARGET, "getpodtarget")
    S_(SECCLASS_DOMAIN, DOMAIN__SETPODTARGET, "setpodtarget")
    S_(SECCLASS_DOMAIN, DOMAIN__SET_MISC_INFO, "set_misc_info")
+   S_(SECCLASS_DOMAIN, DOMAIN__SET_VIRQ_HANDLER, "set_virq_handler")
    S_(SECCLASS_HVM, HVM__SETHVMC, "sethvmc")
    S_(SECCLASS_HVM, HVM__GETHVMC, "gethvmc")
    S_(SECCLASS_HVM, HVM__SETPARAM, "setparam")
diff --git a/xen/xsm/flask/include/av_permissions.h b/xen/xsm/flask/include/av_permissions.h
index 9e55a86..42eaf81 100644
--- a/xen/xsm/flask/include/av_permissions.h
+++ b/xen/xsm/flask/include/av_permissions.h
@@ -61,6 +61,7 @@
 #define DOMAIN__GETPODTARGET                      0x10000000UL
 #define DOMAIN__SETPODTARGET                      0x20000000UL
 #define DOMAIN__SET_MISC_INFO                     0x40000000UL
+#define DOMAIN__SET_VIRQ_HANDLER                  0x80000000UL
 
 #define HVM__SETHVMC                              0x00000001UL
 #define HVM__GETHVMC                              0x00000002UL
-- 
1.7.7.5

Re: [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Jan Beulich]

>>> On 13.01.12 at 00:35, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> +int set_global_virq_handler(struct domain *d, uint32_t virq)
> +{
> +    struct domain *old;
> +
> +    if (virq >= NR_VIRQS)
> +        return -EINVAL;
> +    if (!virq_is_global(virq))
> +        return -EINVAL;
> +
> +    if (global_virq_handlers[virq] == d)
> +        return 0;
> +
> +    if (unlikely(!get_domain(d)))
> +        return -EINVAL;
> +
> +    spin_lock(&global_virq_handlers_lock);
> +    old = global_virq_handlers[virq];
> +    global_virq_handlers[virq] = d;
> +    spin_unlock(&global_virq_handlers_lock);
> +
> +    if (old != NULL)
> +        put_domain(old);
> +
> +    return 0;
> +}
> +
> +static void clear_global_virq_handlers(struct domain *d)
> +{
> +    uint32_t virq;
> +    int put_count = 0;
> +
> +    spin_lock(&global_virq_handlers_lock);
> +
> +    for (virq = 0; virq < NR_VIRQS; virq++) {
> +        if (global_virq_handlers[virq] == d) {
> +            global_virq_handlers[virq] = NULL;
> +            put_count++;
> +        }
> +    }
> +
> +    spin_unlock(&global_virq_handlers_lock);
> +
> +    while (put_count) {
> +        put_domain(d);
> +        put_count--;
> +    }
> +}

Formatting in this entire hunk should be changed to match that of the
rest of the file.

> --- a/xen/include/xsm/xsm.h
> +++ b/xen/include/xsm/xsm.h
> @@ -64,6 +64,7 @@ struct xsm_operations {
>      int (*domain_settime) (struct domain *d);
>      int (*set_target) (struct domain *d, struct domain *e);
>      int (*domctl) (struct domain *d, int cmd);
> +    int (*set_virq_handler) (struct domain *d, int virq);

Here and further down, the 'int' still survived.

Jan

Re: [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Daniel De Graaf]

On 01/13/2012 03:03 AM, Jan Beulich wrote:
>>>> On 13.01.12 at 00:35, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>> +int set_global_virq_handler(struct domain *d, uint32_t virq)
>> +{
>> +    struct domain *old;
>> +
>> +    if (virq >= NR_VIRQS)
>> +        return -EINVAL;
>> +    if (!virq_is_global(virq))
>> +        return -EINVAL;
>> +
>> +    if (global_virq_handlers[virq] == d)
>> +        return 0;
>> +
>> +    if (unlikely(!get_domain(d)))
>> +        return -EINVAL;
>> +
>> +    spin_lock(&global_virq_handlers_lock);
>> +    old = global_virq_handlers[virq];
>> +    global_virq_handlers[virq] = d;
>> +    spin_unlock(&global_virq_handlers_lock);
>> +
>> +    if (old != NULL)
>> +        put_domain(old);
>> +
>> +    return 0;
>> +}
>> +
>> +static void clear_global_virq_handlers(struct domain *d)
>> +{
>> +    uint32_t virq;
>> +    int put_count = 0;
>> +
>> +    spin_lock(&global_virq_handlers_lock);
>> +
>> +    for (virq = 0; virq < NR_VIRQS; virq++) {
>> +        if (global_virq_handlers[virq] == d) {
>> +            global_virq_handlers[virq] = NULL;
>> +            put_count++;
>> +        }
>> +    }
>> +
>> +    spin_unlock(&global_virq_handlers_lock);
>> +
>> +    while (put_count) {
>> +        put_domain(d);
>> +        put_count--;
>> +    }
>> +}
> 
> Formatting in this entire hunk should be changed to match that of the
> rest of the file.
> 
>> --- a/xen/include/xsm/xsm.h
>> +++ b/xen/include/xsm/xsm.h
>> @@ -64,6 +64,7 @@ struct xsm_operations {
>>      int (*domain_settime) (struct domain *d);
>>      int (*set_target) (struct domain *d, struct domain *e);
>>      int (*domctl) (struct domain *d, int cmd);
>> +    int (*set_virq_handler) (struct domain *d, int virq);
> 
> Here and further down, the 'int' still survived.
> 
> Jan
> 

Much of the existing code handling virqs uses int; should I also change
these instances to uint32_t?

Re: [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Jan Beulich]

>>> On 13.01.12 at 14:58, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> On 01/13/2012 03:03 AM, Jan Beulich wrote:
>>>>> On 13.01.12 at 00:35, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>> +int set_global_virq_handler(struct domain *d, uint32_t virq)
>>> +{
>>> +    struct domain *old;
>>> +
>>> +    if (virq >= NR_VIRQS)
>>> +        return -EINVAL;
>>> +    if (!virq_is_global(virq))
>>> +        return -EINVAL;
>>> +
>>> +    if (global_virq_handlers[virq] == d)
>>> +        return 0;
>>> +
>>> +    if (unlikely(!get_domain(d)))
>>> +        return -EINVAL;
>>> +
>>> +    spin_lock(&global_virq_handlers_lock);
>>> +    old = global_virq_handlers[virq];
>>> +    global_virq_handlers[virq] = d;
>>> +    spin_unlock(&global_virq_handlers_lock);
>>> +
>>> +    if (old != NULL)
>>> +        put_domain(old);
>>> +
>>> +    return 0;
>>> +}
>>> +
>>> +static void clear_global_virq_handlers(struct domain *d)
>>> +{
>>> +    uint32_t virq;
>>> +    int put_count = 0;
>>> +
>>> +    spin_lock(&global_virq_handlers_lock);
>>> +
>>> +    for (virq = 0; virq < NR_VIRQS; virq++) {
>>> +        if (global_virq_handlers[virq] == d) {
>>> +            global_virq_handlers[virq] = NULL;
>>> +            put_count++;
>>> +        }
>>> +    }
>>> +
>>> +    spin_unlock(&global_virq_handlers_lock);
>>> +
>>> +    while (put_count) {
>>> +        put_domain(d);
>>> +        put_count--;
>>> +    }
>>> +}
>> 
>> Formatting in this entire hunk should be changed to match that of the
>> rest of the file.
>> 
>>> --- a/xen/include/xsm/xsm.h
>>> +++ b/xen/include/xsm/xsm.h
>>> @@ -64,6 +64,7 @@ struct xsm_operations {
>>>      int (*domain_settime) (struct domain *d);
>>>      int (*set_target) (struct domain *d, struct domain *e);
>>>      int (*domctl) (struct domain *d, int cmd);
>>> +    int (*set_virq_handler) (struct domain *d, int virq);
>> 
>> Here and further down, the 'int' still survived.
>> 
>> Jan
>> 
> 
> Much of the existing code handling virqs uses int; should I also change
> these instances to uint32_t?

That would be nice (if you do, making this a separate patch would be
desirable). Here I'm just asking to not repeat the mistake.

Jan

Re: [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> 
> +static void clear_global_virq_handlers(struct domain *d)
> +{
> +    uint32_t virq;
> +    int put_count = 0;
> +
> +    spin_lock(&global_virq_handlers_lock);
> +
> +    for (virq = 0; virq < NR_VIRQS; virq++) {
> +        if (global_virq_handlers[virq] == d) {
> +            global_virq_handlers[virq] = NULL;

I don't suppose we should rebind to dom0, should we? 

Seems like we are pretty hosed if this ever happens in a non-controlled
manner anyway...

> +            put_count++;
> +        }
> +    }
> +
> +    spin_unlock(&global_virq_handlers_lock);
> +
> +    while (put_count) {
> +        put_domain(d);
> +        put_count--;
> +    }
> +} 

Re: [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Jan Beulich]

>>> On 18.01.12 at 11:39, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
>> 
>> +static void clear_global_virq_handlers(struct domain *d)
>> +{
>> +    uint32_t virq;
>> +    int put_count = 0;
>> +
>> +    spin_lock(&global_virq_handlers_lock);
>> +
>> +    for (virq = 0; virq < NR_VIRQS; virq++) {
>> +        if (global_virq_handlers[virq] == d) {
>> +            global_virq_handlers[virq] = NULL;
> 
> I don't suppose we should rebind to dom0, should we? 

Storing NULL here effectively means re-binding to Dom0.

Jan

Re: [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to other domains

[Ian Campbell]

On Wed, 2012-01-18 at 11:28 +0000, Jan Beulich wrote:
> >>> On 18.01.12 at 11:39, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> >> 
> >> +static void clear_global_virq_handlers(struct domain *d)
> >> +{
> >> +    uint32_t virq;
> >> +    int put_count = 0;
> >> +
> >> +    spin_lock(&global_virq_handlers_lock);
> >> +
> >> +    for (virq = 0; virq < NR_VIRQS; virq++) {
> >> +        if (global_virq_handlers[virq] == d) {
> >> +            global_virq_handlers[virq] = NULL;
> > 
> > I don't suppose we should rebind to dom0, should we? 
> 
> Storing NULL here effectively means re-binding to Dom0.

Oh, good, thanks. In that case:
Acked-by: Ian Campbell <ian.campbell@citrix.com>

Ian.

> 
> Jan
> 

[PATCH 03/18] xen: use XSM instead of IS_PRIV for getdomaininfo

[Daniel De Graaf]

The XEN_DOMCTL_getdomaininfo domctl does not allow manipulation of
domains, only basic information such as size and state, so its use
does not fully justify making a domain privileged. XSM modules can also
provide fine-grained control over what domains are visible to domains
that call getdomaininfo.

If XSM is disabled (either at compile time or by using the dummy XSM
module) then there is no change in behavior: only IS_PRIV domains can
use this domctl. If enabled, the XSM module controls access.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 xen/common/domctl.c |    4 ++++
 xen/xsm/dummy.c     |    2 ++
 2 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/xen/common/domctl.c b/xen/common/domctl.c
index 97c7d53..e1bbc9a 100644
--- a/xen/common/domctl.c
+++ b/xen/common/domctl.c
@@ -263,6 +263,10 @@ long do_domctl(XEN_GUEST_HANDLE(xen_domctl_t) u_domctl)
             return -EPERM;
         break;
     }
+#ifdef XSM_ENABLE
+    case XEN_DOMCTL_getdomaininfo:
+        break;
+#endif
     default:
         if ( !IS_PRIV(current->domain) )
             return -EPERM;
diff --git a/xen/xsm/dummy.c b/xen/xsm/dummy.c
index 59db86d..f94f616 100644
--- a/xen/xsm/dummy.c
+++ b/xen/xsm/dummy.c
@@ -66,6 +66,8 @@ static int dummy_scheduler (struct domain *d)
 
 static int dummy_getdomaininfo (struct domain *d)
 {
+    if ( !IS_PRIV(current->domain) )
+        return -EPERM;
     return 0;
 }
 
-- 
1.7.7.5

[PATCH 04/18] xen: Preserve reserved grant entries when switching versions

[Daniel De Graaf]

In order for the toolstack to use reserved grant table entries, the
grant table for a guest must be initialized prior to the guest's boot.
When the guest switches grant table versions (necessary if the guest is
using v2 grant tables, or on kexec if switching grant versions), these
initial grants will be cleared. Instead of clearing them, preserve
the grants across the type change.

Attempting to use or preserve v2-only features such as sub-page grants
will produce invalid v1 grant entries, which (while they will not work)
is not a problem since a guest can always produce such invalid entries.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 xen/common/grant_table.c         |   38 +++++++++++++++++++++++++++++++-------
 xen/include/public/grant_table.h |    6 ++++++
 2 files changed, 37 insertions(+), 7 deletions(-)

diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
index 014734d..8d4a4cb 100644
--- a/xen/common/grant_table.c
+++ b/xen/common/grant_table.c
@@ -2106,6 +2106,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
     struct domain *d = current->domain;
     struct grant_table *gt = d->grant_table;
     struct active_grant_entry *act;
+    grant_entry_v1_t reserved_entries[GNTTAB_NR_RESERVED_ENTRIES];
     long res;
     int i;
 
@@ -2126,7 +2127,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
     /* (You need to change the version number for e.g. kexec.) */
     if ( gt->gt_version != 0 )
     {
-        for ( i = 0; i < nr_grant_entries(gt); i++ )
+        for ( i = GNTTAB_NR_RESERVED_ENTRIES; i < nr_grant_entries(gt); i++ )
         {
             act = &active_entry(gt, i);
             if ( act->pin != 0 )
@@ -2151,15 +2152,38 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
             goto out_unlock;
     }
 
+    /* Preserve the first 8 entries (toolstack reserved grants) */
+    if (gt->gt_version == 1) {
+        memcpy(reserved_entries, gt->shared_v1[0], sizeof(reserved_entries));
+    } else if (gt->gt_version == 2) {
+        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES && i < nr_grant_entries(gt); i++ )
+        {
+            reserved_entries[i].flags = shared_entry_v2(gt, i).hdr.flags;
+            reserved_entries[i].domid = shared_entry_v2(gt, i).hdr.domid;
+            reserved_entries[i].frame = shared_entry_v2(gt, i).full_page.frame;
+            reserved_entries[i].flags |= status_entry(gt, i);
+        }
+    }
+
     if ( op.version < 2 && gt->gt_version == 2 )
         gnttab_unpopulate_status_frames(d, gt);
 
-    if ( op.version != gt->gt_version )
-    {
-        /* Make sure there's no crud left over in the table from the
-           old version. */
-        for ( i = 0; i < nr_grant_frames(gt); i++ )
-            memset(gt->shared_raw[i], 0, PAGE_SIZE);
+    /* Make sure there's no crud left over in the table from the
+       old version. */
+    for ( i = 0; i < nr_grant_frames(gt); i++ )
+        memset(gt->shared_raw[i], 0, PAGE_SIZE);
+
+    /* Restore the first 8 entries (toolstack reserved grants) */
+    if (gt->gt_version != 0 && op.version == 1) {
+        memcpy(gt->shared_v1[0], reserved_entries, sizeof(reserved_entries));
+    } else if (gt->gt_version != 0 && op.version == 2) {
+        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES; i++ )
+        {
+            status_entry(gt, i) = reserved_entries[i].flags & (GTF_reading|GTF_writing);
+            shared_entry_v2(gt, i).hdr.flags = reserved_entries[i].flags & ~(GTF_reading|GTF_writing);
+            shared_entry_v2(gt, i).hdr.domid = reserved_entries[i].domid;
+            shared_entry_v2(gt, i).full_page.frame = reserved_entries[i].frame;
+        }
     }
 
     gt->gt_version = op.version;
diff --git a/xen/include/public/grant_table.h b/xen/include/public/grant_table.h
index 0bf20bc..36d1ac7 100644
--- a/xen/include/public/grant_table.h
+++ b/xen/include/public/grant_table.h
@@ -117,6 +117,12 @@ struct grant_entry_v1 {
 };
 typedef struct grant_entry_v1 grant_entry_v1_t;
 
+/* External tools reserve first few grant table entries. */
+#define GNTTAB_NR_RESERVED_ENTRIES     8
+#define GNTTAB_RESERVED_CONSOLE        0
+#define GNTTAB_RESERVED_XENSTORE       1
+/* (the next 6 are reserved for future use) */
+
 /*
  * Type of grant entry.
  *  GTF_invalid: This grant entry grants no privileges.
-- 
1.7.7.5

Re: [PATCH 04/18] xen: Preserve reserved grant entries when switching versions

[Jan Beulich]

>>> On 13.01.12 at 00:35, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> --- a/xen/include/public/grant_table.h
> +++ b/xen/include/public/grant_table.h
> @@ -117,6 +117,12 @@ struct grant_entry_v1 {
>  };
>  typedef struct grant_entry_v1 grant_entry_v1_t;
>  
> +/* External tools reserve first few grant table entries. */

This reads as if they must not be used (but could without taking extra
care) by other than the tools themselves, which would be an
incompatible change. As my understanding is that the change is
backwards compatible, I would suggest adjusting the comment.

Jan

> +#define GNTTAB_NR_RESERVED_ENTRIES     8
> +#define GNTTAB_RESERVED_CONSOLE        0
> +#define GNTTAB_RESERVED_XENSTORE       1
> +/* (the next 6 are reserved for future use) */
> +
>  /*
>   * Type of grant entry.
>   *  GTF_invalid: This grant entry grants no privileges.

Re: [PATCH 04/18] xen: Preserve reserved grant entries when switching versions

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> In order for the toolstack to use reserved grant table entries, the
> grant table for a guest must be initialized prior to the guest's boot.
> When the guest switches grant table versions (necessary if the guest is
> using v2 grant tables, or on kexec if switching grant versions), these
> initial grants will be cleared. Instead of clearing them, preserve
> the grants across the type change.
> 
> Attempting to use or preserve v2-only features such as sub-page grants
> will produce invalid v1 grant entries, which (while they will not work)
> is not a problem since a guest can always produce such invalid entries.

Would it be better to explicitly clear such entries? Perhaps we should
even warn since it is effectively disallowed for the toolstack to use v2
features in the reserved entries.

> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  xen/common/grant_table.c         |   38 +++++++++++++++++++++++++++++++-------
>  xen/include/public/grant_table.h |    6 ++++++
>  2 files changed, 37 insertions(+), 7 deletions(-)
> 
> diff --git a/xen/common/grant_table.c b/xen/common/grant_table.c
> index 014734d..8d4a4cb 100644
> --- a/xen/common/grant_table.c
> +++ b/xen/common/grant_table.c
> @@ -2106,6 +2106,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
>      struct domain *d = current->domain;
>      struct grant_table *gt = d->grant_table;
>      struct active_grant_entry *act;
> +    grant_entry_v1_t reserved_entries[GNTTAB_NR_RESERVED_ENTRIES];
>      long res;
>      int i;
>  
> @@ -2126,7 +2127,7 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
>      /* (You need to change the version number for e.g. kexec.) */
>      if ( gt->gt_version != 0 )
>      {
> -        for ( i = 0; i < nr_grant_entries(gt); i++ )
> +        for ( i = GNTTAB_NR_RESERVED_ENTRIES; i < nr_grant_entries(gt); i++ )
>          {
>              act = &active_entry(gt, i);
>              if ( act->pin != 0 )
> @@ -2151,15 +2152,38 @@ gnttab_set_version(XEN_GUEST_HANDLE(gnttab_set_version_t uop))
>              goto out_unlock;
>      }
>  
> +    /* Preserve the first 8 entries (toolstack reserved grants) */
> +    if (gt->gt_version == 1) {
> +        memcpy(reserved_entries, gt->shared_v1[0], sizeof(reserved_entries));
> +    } else if (gt->gt_version == 2) {
> +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES && i < nr_grant_entries(gt); i++ )
> +        {
> +            reserved_entries[i].flags = shared_entry_v2(gt, i).hdr.flags;
> +            reserved_entries[i].domid = shared_entry_v2(gt, i).hdr.domid;
> +            reserved_entries[i].frame = shared_entry_v2(gt, i).full_page.frame;
> +            reserved_entries[i].flags |= status_entry(gt, i);
> +        }
> +    }
> +
>      if ( op.version < 2 && gt->gt_version == 2 )
>          gnttab_unpopulate_status_frames(d, gt);
>  
> -    if ( op.version != gt->gt_version )
> -    {
> -        /* Make sure there's no crud left over in the table from the
> -           old version. */
> -        for ( i = 0; i < nr_grant_frames(gt); i++ )
> -            memset(gt->shared_raw[i], 0, PAGE_SIZE);
> +    /* Make sure there's no crud left over in the table from the
> +       old version. */
> +    for ( i = 0; i < nr_grant_frames(gt); i++ )
> +        memset(gt->shared_raw[i], 0, PAGE_SIZE);
> +
> +    /* Restore the first 8 entries (toolstack reserved grants) */
> +    if (gt->gt_version != 0 && op.version == 1) {
> +        memcpy(gt->shared_v1[0], reserved_entries, sizeof(reserved_entries));
> +    } else if (gt->gt_version != 0 && op.version == 2) {
> +        for ( i = 0; i < GNTTAB_NR_RESERVED_ENTRIES; i++ )
> +        {
> +            status_entry(gt, i) = reserved_entries[i].flags & (GTF_reading|GTF_writing);
> +            shared_entry_v2(gt, i).hdr.flags = reserved_entries[i].flags & ~(GTF_reading|GTF_writing);
> +            shared_entry_v2(gt, i).hdr.domid = reserved_entries[i].domid;
> +            shared_entry_v2(gt, i).full_page.frame = reserved_entries[i].frame;
> +        }
>      }
>  
>      gt->gt_version = op.version;
> diff --git a/xen/include/public/grant_table.h b/xen/include/public/grant_table.h
> index 0bf20bc..36d1ac7 100644
> --- a/xen/include/public/grant_table.h
> +++ b/xen/include/public/grant_table.h
> @@ -117,6 +117,12 @@ struct grant_entry_v1 {
>  };
>  typedef struct grant_entry_v1 grant_entry_v1_t;
>  
> +/* External tools reserve first few grant table entries. */
> +#define GNTTAB_NR_RESERVED_ENTRIES     8
> +#define GNTTAB_RESERVED_CONSOLE        0
> +#define GNTTAB_RESERVED_XENSTORE       1
> +/* (the next 6 are reserved for future use) */
> +
>  /*
>   * Type of grant entry.
>   *  GTF_invalid: This grant entry grants no privileges.

[PATCH 05/18] tools/libxl: pull xenstore/console domids from xenstore

[Daniel De Graaf]

Instead of assuming that xenstored and xenconsoled are running in dom0,
pull the domain IDs from xenstore.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/libxl/libxl_dom.c      |   14 ++++++++++++--
 tools/libxl/libxl_internal.h |    2 ++
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index a4725fe..5e39595 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -73,6 +73,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
 {
     libxl_ctx *ctx = libxl__gc_owner(gc);
     int tsc_mode;
+    char *xs_domid, *con_domid;
     xc_domain_max_vcpus(ctx->xch, domid, info->max_vcpus);
     xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb + LIBXL_MAXMEM_CONSTANT);
     if (info->type == LIBXL_DOMAIN_TYPE_PV)
@@ -104,9 +105,18 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
         xc_shadow_control(ctx->xch, domid, XEN_DOMCTL_SHADOW_OP_SET_ALLOCATION, NULL, 0, &shadow, 0, NULL);
     }
 
-    state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, 0);
-    state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, 0);
+    xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
+    state->store_domid = xs_domid ? atoi(xs_domid) : 0;
+    free(xs_domid);
+
+    con_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenconsoled/domid", NULL);
+    state->console_domid = con_domid ? atoi(con_domid) : 0;
+    free(con_domid);
+
+    state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, state->store_domid);
+    state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, state->console_domid);
     state->vm_generationid_addr = 0;
+
     return 0;
 }
 
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 288c03c..97ead08 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -219,9 +219,11 @@ _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
     libxl__domain_type((gc), (domid)) == LIBXL_DOMAIN_TYPE_##type
 typedef struct {
     uint32_t store_port;
+    uint32_t store_domid;
     unsigned long store_mfn;
 
     uint32_t console_port;
+    uint32_t console_domid;
     unsigned long console_mfn;
     unsigned long vm_generationid_addr;
 } libxl__domain_build_state;
-- 
1.7.7.5

Re: [PATCH 05/18] tools/libxl: pull xenstore/console domids from xenstore

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> Instead of assuming that xenstored and xenconsoled are running in dom0,
> pull the domain IDs from xenstore.
> 
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

I'm not sure we need to carry these around in the build state, since
they can just be looked up (perhaps by helper functions) but
nevertheless:

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_dom.c      |   14 ++++++++++++--
>  tools/libxl/libxl_internal.h |    2 ++
>  2 files changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index a4725fe..5e39595 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
> @@ -73,6 +73,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
>  {
>      libxl_ctx *ctx = libxl__gc_owner(gc);
>      int tsc_mode;
> +    char *xs_domid, *con_domid;
>      xc_domain_max_vcpus(ctx->xch, domid, info->max_vcpus);
>      xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb + LIBXL_MAXMEM_CONSTANT);
>      if (info->type == LIBXL_DOMAIN_TYPE_PV)
> @@ -104,9 +105,18 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
>          xc_shadow_control(ctx->xch, domid, XEN_DOMCTL_SHADOW_OP_SET_ALLOCATION, NULL, 0, &shadow, 0, NULL);
>      }
>  
> -    state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, 0);
> -    state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, 0);
> +    xs_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenstored/domid", NULL);
> +    state->store_domid = xs_domid ? atoi(xs_domid) : 0;
> +    free(xs_domid);
> +
> +    con_domid = xs_read(ctx->xsh, XBT_NULL, "/tool/xenconsoled/domid", NULL);
> +    state->console_domid = con_domid ? atoi(con_domid) : 0;
> +    free(con_domid);
> +
> +    state->store_port = xc_evtchn_alloc_unbound(ctx->xch, domid, state->store_domid);
> +    state->console_port = xc_evtchn_alloc_unbound(ctx->xch, domid, state->console_domid);
>      state->vm_generationid_addr = 0;
> +
>      return 0;
>  }
>  
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 288c03c..97ead08 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -219,9 +219,11 @@ _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
>      libxl__domain_type((gc), (domid)) == LIBXL_DOMAIN_TYPE_##type
>  typedef struct {
>      uint32_t store_port;
> +    uint32_t store_domid;
>      unsigned long store_mfn;
>  
>      uint32_t console_port;
> +    uint32_t console_domid;
>      unsigned long console_mfn;
>      unsigned long vm_generationid_addr;
>  } libxl__domain_build_state;

[PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

This patch claims one reserved grant entry for the console and another
for the xenstore. It modifies the builder to fill in the grant table
entries for the console and the xenstore.

Previous versions of this patch have been sent to xen-devel. See
http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00610.html
http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01491.html

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/libxc/xc_dom.h              |   13 +++
 tools/libxc/xc_dom_boot.c         |  164 +++++++++++++++++++++++++++++++++++++
 tools/libxc/xc_dom_compat_linux.c |    2 +
 tools/libxc/xc_domain_restore.c   |   19 ++++-
 tools/libxc/xenguest.h            |    3 +-
 tools/libxl/libxl_dom.c           |   18 ++++-
 tools/xcutils/xc_restore.c        |    4 +-
 7 files changed, 216 insertions(+), 7 deletions(-)

diff --git a/tools/libxc/xc_dom.h b/tools/libxc/xc_dom.h
index e72f066..6c36403 100644
--- a/tools/libxc/xc_dom.h
+++ b/tools/libxc/xc_dom.h
@@ -106,7 +106,9 @@ struct xc_dom_image {
     /* misc xen domain config stuff */
     unsigned long flags;
     unsigned int console_evtchn;
+    unsigned int console_domid;
     unsigned int xenstore_evtchn;
+    unsigned int xenstore_domid;
     xen_pfn_t shared_info_mfn;
 
     xc_interface *xch;
@@ -200,6 +202,17 @@ void *xc_dom_boot_domU_map(struct xc_dom_image *dom, xen_pfn_t pfn,
                            xen_pfn_t count);
 int xc_dom_boot_image(struct xc_dom_image *dom);
 int xc_dom_compat_check(struct xc_dom_image *dom);
+int xc_dom_gnttab_init(struct xc_dom_image *dom);
+int xc_dom_gnttab_hvm_seed(xc_interface *xch, uint32_t domid,
+                           unsigned long console_gmfn,
+                           unsigned long xenstore_gmfn,
+                           uint32_t console_domid,
+                           uint32_t xenstore_domid);
+int xc_dom_gnttab_seed(xc_interface *xch, uint32_t domid,
+                       unsigned long console_gmfn,
+                       unsigned long xenstore_gmfn,
+                       uint32_t console_domid,
+                       uint32_t xenstore_domid);
 
 /* --- debugging bits ---------------------------------------------- */
 
diff --git a/tools/libxc/xc_dom_boot.c b/tools/libxc/xc_dom_boot.c
index 65f60df..1a55a6d 100644
--- a/tools/libxc/xc_dom_boot.c
+++ b/tools/libxc/xc_dom_boot.c
@@ -34,6 +34,7 @@
 #include "xg_private.h"
 #include "xc_dom.h"
 #include <xen/hvm/params.h>
+#include <xen/grant_table.h>
 
 /* ------------------------------------------------------------------------ */
 
@@ -275,6 +276,169 @@ int xc_dom_boot_image(struct xc_dom_image *dom)
     return rc;
 }
 
+static unsigned long xc_dom_gnttab_setup(xc_interface *xch, uint32_t domid)
+{
+    DECLARE_HYPERCALL;
+    gnttab_setup_table_t setup_table;
+    DECLARE_HYPERCALL_BUFFER(unsigned long, gmfnp);
+    int rc;
+	unsigned long gmfn;
+
+    gmfnp = xc_hypercall_buffer_alloc(xch, gmfnp, sizeof(*gmfnp));
+	if (gmfnp == NULL)
+		return -1;
+
+    setup_table.dom = domid;
+    setup_table.nr_frames = 1;
+    set_xen_guest_handle(setup_table.frame_list, gmfnp);
+    setup_table.status = 0;
+
+    hypercall.op = __HYPERVISOR_grant_table_op;
+    hypercall.arg[0] = GNTTABOP_setup_table;
+    hypercall.arg[1] = (unsigned long) &setup_table;
+    hypercall.arg[2] = 1;
+
+    rc = do_xen_hypercall(xch, &hypercall);
+	gmfn = *gmfnp;
+    xc_hypercall_buffer_free(xch, gmfnp);
+
+    if ( rc != 0 || setup_table.status != GNTST_okay )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to setup domU grant table "
+                     "[errno=%d, status=%" PRId16 "]\n",
+                     __FUNCTION__, rc != 0 ? errno : 0, setup_table.status);
+        return -1;
+    }
+
+    return gmfn;
+}
+
+int xc_dom_gnttab_seed(xc_interface *xch, uint32_t domid,
+                       unsigned long console_gmfn,
+                       unsigned long xenstore_gmfn,
+                       uint32_t console_domid,
+                       uint32_t xenstore_domid)
+{
+
+    unsigned long gnttab_gmfn;
+    grant_entry_v1_t *gnttab;
+
+    gnttab_gmfn = xc_dom_gnttab_setup(xch, domid);
+    if ( gnttab_gmfn == -1 )
+        return -1;
+
+    gnttab = xc_map_foreign_range(xch,
+                                  domid,
+                                  PAGE_SIZE,
+                                  PROT_READ|PROT_WRITE,
+                                  gnttab_gmfn);
+    if ( gnttab == NULL )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to map domU grant table "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    if ( domid != console_domid  && console_gmfn != -1)
+    {
+        gnttab[GNTTAB_RESERVED_CONSOLE].flags = GTF_permit_access;
+        gnttab[GNTTAB_RESERVED_CONSOLE].domid = console_domid;
+        gnttab[GNTTAB_RESERVED_CONSOLE].frame = console_gmfn;
+    }
+    if ( domid != xenstore_domid && xenstore_gmfn != -1)
+    {
+        gnttab[GNTTAB_RESERVED_XENSTORE].flags = GTF_permit_access;
+        gnttab[GNTTAB_RESERVED_XENSTORE].domid = xenstore_domid;
+        gnttab[GNTTAB_RESERVED_XENSTORE].frame = xenstore_gmfn;
+    }
+
+    if ( munmap(gnttab, PAGE_SIZE) == -1 )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to unmap domU grant table "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    return 0;
+}
+
+int xc_dom_gnttab_hvm_seed(xc_interface *xch, uint32_t domid,
+                           unsigned long console_gpfn,
+                           unsigned long xenstore_gpfn,
+                           uint32_t console_domid,
+                           uint32_t xenstore_domid)
+{
+#define SCRATCH_PFN_GNTTAB 0xFFFFE
+
+    int rc;
+    struct xen_add_to_physmap xatp = {
+        .domid = domid,
+        .space = XENMAPSPACE_grant_table,
+        .idx   = 0, /* TODO: what's this? */
+        .gpfn  = SCRATCH_PFN_GNTTAB
+    };
+    struct xen_remove_from_physmap xrfp = {
+    .domid = domid,
+    .gpfn  = SCRATCH_PFN_GNTTAB
+    };
+
+    rc = do_memory_op(xch, XENMEM_add_to_physmap, &xatp, sizeof(xatp));
+    if ( rc != 0 )
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to add gnttab to physmap "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    rc = xc_dom_gnttab_seed(xch, domid,
+                            console_gpfn, xenstore_gpfn,
+                            console_domid, xenstore_domid);
+    if (rc != 0)
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to seed gnttab entries\n",
+                     __FUNCTION__);
+        (void) do_memory_op(xch, XENMEM_remove_from_physmap, &xrfp, sizeof(xrfp));
+        return -1;
+    }
+
+    rc = do_memory_op(xch, XENMEM_remove_from_physmap, &xrfp, sizeof(xrfp));
+    if (rc != 0)
+    {
+        xc_dom_panic(xch, XC_INTERNAL_ERROR,
+                     "%s: failed to remove gnttab from physmap "
+                     "[errno=%d]\n",
+                     __FUNCTION__, errno);
+        return -1;
+    }
+
+    return 0;
+}
+
+int xc_dom_gnttab_init(struct xc_dom_image *dom)
+{
+    unsigned long console_gmfn;
+    unsigned long xenstore_gmfn;
+    int autotranslated;
+
+    autotranslated = xc_dom_feature_translated(dom);
+    console_gmfn = autotranslated ?
+           dom->console_pfn : xc_dom_p2m_host(dom, dom->console_pfn);
+    xenstore_gmfn = autotranslated ?
+           dom->xenstore_pfn : xc_dom_p2m_host(dom, dom->xenstore_pfn);
+
+    return xc_dom_gnttab_seed(dom->xch, dom->guest_domid,
+                              console_gmfn, xenstore_gmfn,
+                              dom->console_domid, dom->xenstore_domid);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxc/xc_dom_compat_linux.c b/tools/libxc/xc_dom_compat_linux.c
index 0e78842..2183a3b 100644
--- a/tools/libxc/xc_dom_compat_linux.c
+++ b/tools/libxc/xc_dom_compat_linux.c
@@ -62,6 +62,8 @@ static int xc_linux_build_internal(struct xc_dom_image *dom,
         goto out;
     if ( (rc = xc_dom_boot_image(dom)) != 0 )
         goto out;
+    if ( (rc = xc_dom_gnttab_init(dom)) != 0)
+        goto out;
 
     *console_mfn = xc_dom_p2m_host(dom, dom->console_pfn);
     *store_mfn = xc_dom_p2m_host(dom, dom->xenstore_pfn);
diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
index 3fda6f8..8bee684 100644
--- a/tools/libxc/xc_domain_restore.c
+++ b/tools/libxc/xc_domain_restore.c
@@ -1259,7 +1259,8 @@ static int apply_batch(xc_interface *xch, uint32_t dom, struct restore_ctx *ctx,
 
 int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
                       unsigned int store_evtchn, unsigned long *store_mfn,
-                      unsigned int console_evtchn, unsigned long *console_mfn,
+                      uint32_t store_domid, unsigned int console_evtchn,
+                      unsigned long *console_mfn, uint32_t console_domid,
                       unsigned int hvm, unsigned int pae, int superpages,
                       int no_incr_generationid,
                       unsigned long *vm_generationid_addr)
@@ -2018,6 +2019,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
         memcpy(ctx->live_p2m, ctx->p2m, dinfo->p2m_size * sizeof(xen_pfn_t));
     munmap(ctx->live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);
 
+    rc = xc_dom_gnttab_seed(xch, dom, *console_mfn, *store_mfn,
+                            console_domid, store_domid);
+    if (rc != 0)
+    {
+        ERROR("error seeding grant table");
+        goto out;
+    }
+
     DPRINTF("Domain ready to be built.\n");
     rc = 0;
     goto out;
@@ -2076,6 +2085,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
         goto out;
     }
 
+    rc = xc_dom_gnttab_hvm_seed(xch, dom, *console_mfn, *store_mfn,
+                                console_domid, store_domid);
+    if (rc != 0)
+    {
+        ERROR("error seeding grant table");
+        goto out;
+    }
+
     /* HVM success! */
     rc = 0;
 
diff --git a/tools/libxc/xenguest.h b/tools/libxc/xenguest.h
index 6026370..3bd5549 100644
--- a/tools/libxc/xenguest.h
+++ b/tools/libxc/xenguest.h
@@ -79,7 +79,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter
  */
 int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
                       unsigned int store_evtchn, unsigned long *store_mfn,
-                      unsigned int console_evtchn, unsigned long *console_mfn,
+                      uint32_t store_domid, unsigned int console_evtchn,
+                      unsigned long *console_mfn, uint32_t console_domid,
                       unsigned int hvm, unsigned int pae, int superpages,
                       int no_incr_generationid,
 		      unsigned long *vm_generationid_addr);
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 5e39595..04db22f 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -233,7 +233,9 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
 
     dom->flags = flags;
     dom->console_evtchn = state->console_port;
+    dom->console_domid = state->console_domid;
     dom->xenstore_evtchn = state->store_port;
+    dom->xenstore_domid = state->store_domid;
 
     if ( (ret = xc_dom_boot_xen_init(dom, ctx->xch, domid)) != 0 ) {
         LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "xc_dom_boot_xen_init failed");
@@ -259,6 +261,10 @@ int libxl__build_pv(libxl__gc *gc, uint32_t domid,
         LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "xc_dom_boot_image failed");
         goto out;
     }
+    if ( (ret = xc_dom_gnttab_init(dom)) != 0 ) {
+        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "xc_dom_gnttab_init failed");
+        goto out;
+    }
 
     state->console_mfn = xc_dom_p2m_host(dom, dom->console_pfn);
     state->store_mfn = xc_dom_p2m_host(dom, dom->xenstore_pfn);
@@ -272,7 +278,8 @@ out:
 static int hvm_build_set_params(xc_interface *handle, uint32_t domid,
                                 libxl_domain_build_info *info,
                                 int store_evtchn, unsigned long *store_mfn,
-                                int console_evtchn, unsigned long *console_mfn)
+                                int console_evtchn, unsigned long *console_mfn,
+                                uint32_t store_domid, uint32_t console_domid)
 {
     struct hvm_info_table *va_hvm;
     uint8_t *va_map, sum;
@@ -305,6 +312,8 @@ static int hvm_build_set_params(xc_interface *handle, uint32_t domid,
     xc_set_hvm_param(handle, domid, HVM_PARAM_NESTEDHVM, info->u.hvm.nested_hvm);
     xc_set_hvm_param(handle, domid, HVM_PARAM_STORE_EVTCHN, store_evtchn);
     xc_set_hvm_param(handle, domid, HVM_PARAM_CONSOLE_EVTCHN, console_evtchn);
+
+    xc_dom_gnttab_hvm_seed(handle, domid, *console_mfn, *store_mfn, console_domid, store_domid);
     return 0;
 }
 
@@ -358,7 +367,9 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
         goto out;
     }
     ret = hvm_build_set_params(ctx->xch, domid, info, state->store_port,
-                               &state->store_mfn, state->console_port, &state->console_mfn);
+                               &state->store_mfn, state->console_port,
+                               &state->console_mfn, state->store_domid,
+                               state->console_domid);
     if (ret) {
         LIBXL__LOG_ERRNOVAL(ctx, LIBXL__LOG_ERROR, ret, "hvm build set params failed");
         goto out;
@@ -396,7 +407,8 @@ int libxl__domain_restore_common(libxl__gc *gc, uint32_t domid,
     }
     rc = xc_domain_restore(ctx->xch, fd, domid,
                            state->store_port, &state->store_mfn,
-                           state->console_port, &state->console_mfn,
+                           state->store_domid, state->console_port,
+                           &state->console_mfn, state->console_domid,
                            hvm, pae, superpages, no_incr_generationid,
                            &state->vm_generationid_addr);
     if ( rc ) {
diff --git a/tools/xcutils/xc_restore.c b/tools/xcutils/xc_restore.c
index 63d53a8..e41a133 100644
--- a/tools/xcutils/xc_restore.c
+++ b/tools/xcutils/xc_restore.c
@@ -45,8 +45,8 @@ main(int argc, char **argv)
     else
 	    superpages = !!hvm;
 
-    ret = xc_domain_restore(xch, io_fd, domid, store_evtchn, &store_mfn,
-                            console_evtchn, &console_mfn, hvm, pae, superpages,
+    ret = xc_domain_restore(xch, io_fd, domid, store_evtchn, &store_mfn, 0,
+                            console_evtchn, &console_mfn, 0, hvm, pae, superpages,
                             0, NULL);
 
     if ( ret == 0 )
-- 
1.7.7.5

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> @@ -275,6 +276,169 @@ int xc_dom_boot_image(struct xc_dom_image *dom)
>      return rc;
>  }
> 
> +static unsigned long xc_dom_gnttab_setup(xc_interface *xch, uint32_t domid)
> +{
> +    DECLARE_HYPERCALL;
> +    gnttab_setup_table_t setup_table;
> +    DECLARE_HYPERCALL_BUFFER(unsigned long, gmfnp);
> +    int rc;
> +       unsigned long gmfn;

There's a bunch of weird alignment issues in this patch. Presumably some
hard tabs have snuck in.

> +
> +    gmfnp = xc_hypercall_buffer_alloc(xch, gmfnp, sizeof(*gmfnp));
> +       if (gmfnp == NULL)
> +               return -1;

Alignment.

> +
> +    setup_table.dom = domid;
> +    setup_table.nr_frames = 1;
> +    set_xen_guest_handle(setup_table.frame_list, gmfnp);
> +    setup_table.status = 0;
> +
> +    hypercall.op = __HYPERVISOR_grant_table_op;
> +    hypercall.arg[0] = GNTTABOP_setup_table;
> +    hypercall.arg[1] = (unsigned long) &setup_table;
> +    hypercall.arg[2] = 1;
> +
> +    rc = do_xen_hypercall(xch, &hypercall);
> +       gmfn = *gmfnp;

Alignment. I'm going to stop pointing these out, I guess grep will find
them...

[...]
> +int xc_dom_gnttab_init(struct xc_dom_image *dom)
> +{
> +    unsigned long console_gmfn;
> +    unsigned long xenstore_gmfn;
> +    int autotranslated;
> +
> +    autotranslated = xc_dom_feature_translated(dom);
> +    console_gmfn = autotranslated ?
> +           dom->console_pfn : xc_dom_p2m_host(dom, dom->console_pfn);
> +    xenstore_gmfn = autotranslated ?
> +           dom->xenstore_pfn : xc_dom_p2m_host(dom, dom->xenstore_pfn);
> +
> +    return xc_dom_gnttab_seed(dom->xch, dom->guest_domid,
> +                              console_gmfn, xenstore_gmfn,
> +                              dom->console_domid, dom->xenstore_domid);

So on build we have can do the p2m lookup and hence use the PV version
of gnttab_seed, whereas on restore we don't have a p2m and hence can't?

The internals of xc_domain_restore do have the p2m though, so in
principal we could avoid the need for the hvm version and the
reintroduction of the remove_from_physmap by making xc_domain_restore
output the necessary mfns? I'm not sure it is worth it though.


> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
> index 3fda6f8..8bee684 100644
> --- a/tools/libxc/xc_domain_restore.c
> +++ b/tools/libxc/xc_domain_restore.c
> @@ -1259,7 +1259,8 @@ static int apply_batch(xc_interface *xch, uint32_t dom, struct restore_ctx *ctx,
> 
>  int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>                        unsigned int store_evtchn, unsigned long *store_mfn,
> -                      unsigned int console_evtchn, unsigned long *console_mfn,
> +                      uint32_t store_domid, unsigned int console_evtchn,
> +                      unsigned long *console_mfn, uint32_t console_domid,
>                        unsigned int hvm, unsigned int pae, int superpages,
>                        int no_incr_generationid,
>                        unsigned long *vm_generationid_addr)
> @@ -2018,6 +2019,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>          memcpy(ctx->live_p2m, ctx->p2m, dinfo->p2m_size * sizeof(xen_pfn_t));
>      munmap(ctx->live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);
> 
> +    rc = xc_dom_gnttab_seed(xch, dom, *console_mfn, *store_mfn,
> +                            console_domid, store_domid);
> +    if (rc != 0)
> +    {
> +        ERROR("error seeding grant table");
> +        goto out;
> +    }
> +
>      DPRINTF("Domain ready to be built.\n");
>      rc = 0;
>      goto out;
> @@ -2076,6 +2085,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>          goto out;
>      }
> 
> +    rc = xc_dom_gnttab_hvm_seed(xch, dom, *console_mfn, *store_mfn,
> +                                console_domid, store_domid);

Oh, we don't even need to return them from xc_domain_restore since we
could just translate from gfn to mfn right here (since we have a p2m in
hand) and call xc_dom_gnttab_seed. Or am I missing something?

> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index 5e39595..04db22f 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
[...]
> @@ -305,6 +312,8 @@ static int hvm_build_set_params(xc_interface *handle, uint32_t domid,
>      xc_set_hvm_param(handle, domid, HVM_PARAM_NESTEDHVM, info->u.hvm.nested_hvm);
>      xc_set_hvm_param(handle, domid, HVM_PARAM_STORE_EVTCHN, store_evtchn);
>      xc_set_hvm_param(handle, domid, HVM_PARAM_CONSOLE_EVTCHN, console_evtchn);
> +
> +    xc_dom_gnttab_hvm_seed(handle, domid, *console_mfn, *store_mfn, console_domid, store_domid);

Didn't this already happen in xc_linux_build_internal which was called
via xc_linux_build_mem?

[...]
Ian.

Re: [PATCH 06/18] lib{xc, xl}: Seed grant tables with xenstore and console grants

[Daniel De Graaf]

On 01/18/2012 06:05 AM, Ian Campbell wrote:
> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
>> @@ -275,6 +276,169 @@ int xc_dom_boot_image(struct xc_dom_image *dom)
>>      return rc;
>>  }
>>
>> +static unsigned long xc_dom_gnttab_setup(xc_interface *xch, uint32_t domid)
>> +{
>> +    DECLARE_HYPERCALL;
>> +    gnttab_setup_table_t setup_table;
>> +    DECLARE_HYPERCALL_BUFFER(unsigned long, gmfnp);
>> +    int rc;
>> +       unsigned long gmfn;
> 
> There's a bunch of weird alignment issues in this patch. Presumably some
> hard tabs have snuck in.
> 
>> +
>> +    gmfnp = xc_hypercall_buffer_alloc(xch, gmfnp, sizeof(*gmfnp));
>> +       if (gmfnp == NULL)
>> +               return -1;
> 
> Alignment.
> 
>> +
>> +    setup_table.dom = domid;
>> +    setup_table.nr_frames = 1;
>> +    set_xen_guest_handle(setup_table.frame_list, gmfnp);
>> +    setup_table.status = 0;
>> +
>> +    hypercall.op = __HYPERVISOR_grant_table_op;
>> +    hypercall.arg[0] = GNTTABOP_setup_table;
>> +    hypercall.arg[1] = (unsigned long) &setup_table;
>> +    hypercall.arg[2] = 1;
>> +
>> +    rc = do_xen_hypercall(xch, &hypercall);
>> +       gmfn = *gmfnp;
> 
> Alignment. I'm going to stop pointing these out, I guess grep will find
> them...
> 
> [...]
>> +int xc_dom_gnttab_init(struct xc_dom_image *dom)
>> +{
>> +    unsigned long console_gmfn;
>> +    unsigned long xenstore_gmfn;
>> +    int autotranslated;
>> +
>> +    autotranslated = xc_dom_feature_translated(dom);
>> +    console_gmfn = autotranslated ?
>> +           dom->console_pfn : xc_dom_p2m_host(dom, dom->console_pfn);
>> +    xenstore_gmfn = autotranslated ?
>> +           dom->xenstore_pfn : xc_dom_p2m_host(dom, dom->xenstore_pfn);
>> +
>> +    return xc_dom_gnttab_seed(dom->xch, dom->guest_domid,
>> +                              console_gmfn, xenstore_gmfn,
>> +                              dom->console_domid, dom->xenstore_domid);
> 
> So on build we have can do the p2m lookup and hence use the PV version
> of gnttab_seed, whereas on restore we don't have a p2m and hence can't?
> 
> The internals of xc_domain_restore do have the p2m though, so in
> principal we could avoid the need for the hvm version and the
> reintroduction of the remove_from_physmap by making xc_domain_restore
> output the necessary mfns? I'm not sure it is worth it though.
> 

The reason for needing remove_from_physmap is to temporarily access the
grant table, which can be done differently in PV and is not related to
p2m translation.

> 
>> diff --git a/tools/libxc/xc_domain_restore.c b/tools/libxc/xc_domain_restore.c
>> index 3fda6f8..8bee684 100644
>> --- a/tools/libxc/xc_domain_restore.c
>> +++ b/tools/libxc/xc_domain_restore.c
>> @@ -1259,7 +1259,8 @@ static int apply_batch(xc_interface *xch, uint32_t dom, struct restore_ctx *ctx,
>>
>>  int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>>                        unsigned int store_evtchn, unsigned long *store_mfn,
>> -                      unsigned int console_evtchn, unsigned long *console_mfn,
>> +                      uint32_t store_domid, unsigned int console_evtchn,
>> +                      unsigned long *console_mfn, uint32_t console_domid,
>>                        unsigned int hvm, unsigned int pae, int superpages,
>>                        int no_incr_generationid,
>>                        unsigned long *vm_generationid_addr)
>> @@ -2018,6 +2019,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>>          memcpy(ctx->live_p2m, ctx->p2m, dinfo->p2m_size * sizeof(xen_pfn_t));
>>      munmap(ctx->live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);
>>
>> +    rc = xc_dom_gnttab_seed(xch, dom, *console_mfn, *store_mfn,
>> +                            console_domid, store_domid);
>> +    if (rc != 0)
>> +    {
>> +        ERROR("error seeding grant table");
>> +        goto out;
>> +    }
>> +
>>      DPRINTF("Domain ready to be built.\n");
>>      rc = 0;
>>      goto out;
>> @@ -2076,6 +2085,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
>>          goto out;
>>      }
>>
>> +    rc = xc_dom_gnttab_hvm_seed(xch, dom, *console_mfn, *store_mfn,
>> +                                console_domid, store_domid);
> 
> Oh, we don't even need to return them from xc_domain_restore since we
> could just translate from gfn to mfn right here (since we have a p2m in
> hand) and call xc_dom_gnttab_seed. Or am I missing something?

For HVM, we still have to use guest frame numbers because that's what Xen
expects to find in the grant table.

Or are you talking about the fact xc_domain_restore returns the mfns? That
isn't added by this patch; they are just used to populate the grant table.
 
>> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
>> index 5e39595..04db22f 100644
>> --- a/tools/libxl/libxl_dom.c
>> +++ b/tools/libxl/libxl_dom.c
> [...]
>> @@ -305,6 +312,8 @@ static int hvm_build_set_params(xc_interface *handle, uint32_t domid,
>>      xc_set_hvm_param(handle, domid, HVM_PARAM_NESTEDHVM, info->u.hvm.nested_hvm);
>>      xc_set_hvm_param(handle, domid, HVM_PARAM_STORE_EVTCHN, store_evtchn);
>>      xc_set_hvm_param(handle, domid, HVM_PARAM_CONSOLE_EVTCHN, console_evtchn);
>> +
>> +    xc_dom_gnttab_hvm_seed(handle, domid, *console_mfn, *store_mfn, console_domid, store_domid);
> 
> Didn't this already happen in xc_linux_build_internal which was called
> via xc_linux_build_mem?

No, xl doesn't call xc_linux_build_mem; that's only used by xm.

-- 
Daniel De Graaf
National Security Agency

[PATCH 07/18] mini-os: avoid crash if no console is provided

[Daniel De Graaf]

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/console/xencons_ring.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/extras/mini-os/console/xencons_ring.c b/extras/mini-os/console/xencons_ring.c
index 22fd618..14a8bd1 100644
--- a/extras/mini-os/console/xencons_ring.c
+++ b/extras/mini-os/console/xencons_ring.c
@@ -25,7 +25,10 @@ static inline void notify_daemon(struct consfront_dev *dev)
 
 static inline struct xencons_interface *xencons_interface(void)
 {
-    return mfn_to_virt(start_info.console.domU.mfn);
+    if (start_info.console.domU.evtchn)
+        return mfn_to_virt(start_info.console.domU.mfn);
+    else
+        return NULL;
 } 
  
 int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, unsigned len)
@@ -38,6 +41,8 @@ int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, uns
             intf = xencons_interface();
         else
             intf = dev->ring;
+    if (!intf)
+        return sent;
 
 	cons = intf->out_cons;
 	prod = intf->out_prod;
-- 
1.7.7.5

Re: [PATCH 07/18] mini-os: avoid crash if no console is provided

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  extras/mini-os/console/xencons_ring.c |    7 ++++++-
>  1 files changed, 6 insertions(+), 1 deletions(-)
> 
> diff --git a/extras/mini-os/console/xencons_ring.c b/extras/mini-os/console/xencons_ring.c
> index 22fd618..14a8bd1 100644
> --- a/extras/mini-os/console/xencons_ring.c
> +++ b/extras/mini-os/console/xencons_ring.c
> @@ -25,7 +25,10 @@ static inline void notify_daemon(struct consfront_dev *dev)
>  
>  static inline struct xencons_interface *xencons_interface(void)
>  {
> -    return mfn_to_virt(start_info.console.domU.mfn);
> +    if (start_info.console.domU.evtchn)
> +        return mfn_to_virt(start_info.console.domU.mfn);
> +    else
> +        return NULL;
>  } 
>   
>  int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, unsigned len)
> @@ -38,6 +41,8 @@ int xencons_ring_send_no_notify(struct consfront_dev *dev, const char *data, uns
>              intf = xencons_interface();
>          else
>              intf = dev->ring;
> +    if (!intf)
> +        return sent;

Indentation. Otherwise

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Although I would still prefer if we could figure out some scheme to get
logs out of this domain...

Ian.

[PATCH 08/18] mini-os: avoid crash if no xenstore is provided

[Daniel De Graaf]

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/xenbus/xenbus.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/extras/mini-os/xenbus/xenbus.c b/extras/mini-os/xenbus/xenbus.c
index a8081fd..e475e2c 100644
--- a/extras/mini-os/xenbus/xenbus.c
+++ b/extras/mini-os/xenbus/xenbus.c
@@ -328,6 +328,10 @@ static int allocate_xenbus_id(void)
 void init_xenbus(void)
 {
     int err;
+    if (!start_info.store_evtchn) {
+        printk("Skipping initialization of xenbus\n");
+        return;
+    }
     printk("Initialising xenbus\n");
     DEBUG("init_xenbus called.\n");
     xenstore_buf = mfn_to_virt(start_info.store_mfn);
@@ -435,6 +439,11 @@ xenbus_msg_reply(int type,
     DEFINE_WAIT(w);
     struct xsd_sockmsg *rep;
 
+    if (!xenstore_buf) {
+        printk("xenbus_msg_reply called but no xenstore!\n");
+        return NULL;
+    }
+
     id = allocate_xenbus_id();
     add_waiter(w, req_info[id].waitq);
 
-- 
1.7.7.5

Re: [PATCH 08/18] mini-os: avoid crash if no xenstore is provided

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

I suppose we could arrange for xenbus to be omitted entirely but this is
easier.

Ian.

> ---
>  extras/mini-os/xenbus/xenbus.c |    9 +++++++++
>  1 files changed, 9 insertions(+), 0 deletions(-)
> 
> diff --git a/extras/mini-os/xenbus/xenbus.c b/extras/mini-os/xenbus/xenbus.c
> index a8081fd..e475e2c 100644
> --- a/extras/mini-os/xenbus/xenbus.c
> +++ b/extras/mini-os/xenbus/xenbus.c
> @@ -328,6 +328,10 @@ static int allocate_xenbus_id(void)
>  void init_xenbus(void)
>  {
>      int err;
> +    if (!start_info.store_evtchn) {
> +        printk("Skipping initialization of xenbus\n");
> +        return;
> +    }
>      printk("Initialising xenbus\n");
>      DEBUG("init_xenbus called.\n");
>      xenstore_buf = mfn_to_virt(start_info.store_mfn);
> @@ -435,6 +439,11 @@ xenbus_msg_reply(int type,
>      DEFINE_WAIT(w);
>      struct xsd_sockmsg *rep;
>  
> +    if (!xenstore_buf) {
> +        printk("xenbus_msg_reply called but no xenstore!\n");
> +        return NULL;
> +    }
> +
>      id = allocate_xenbus_id();
>      add_waiter(w, req_info[id].waitq);
>  

[PATCH 09/18] mini-os: remove per-fd evtchn limit

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

Changes the minios evtchn implementation to use a list instead of an array.
This allows it to grow as necessary to support any number of ports.
Unfortunately, it's still limited by NR_EVS in events.c.

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/include/lib.h |   16 +++---
 tools/libxc/xc_minios.c      |  139 ++++++++++++++++++++++--------------------
 2 files changed, 81 insertions(+), 74 deletions(-)

diff --git a/extras/mini-os/include/lib.h b/extras/mini-os/include/lib.h
index bd3eeaf..12070c3 100644
--- a/extras/mini-os/include/lib.h
+++ b/extras/mini-os/include/lib.h
@@ -53,6 +53,7 @@
 #include <xen/xen.h>
 #include <xen/event_channel.h>
 #include "gntmap.h"
+#include "list.h"
 
 #ifdef HAVE_LIBC
 #include <stdio.h>
@@ -143,7 +144,12 @@ enum fd_type {
     FTYPE_SAVEFILE,
 };
 
-#define MAX_EVTCHN_PORTS 16
+struct evtchn_port_info {
+        struct minios_list_head list;
+        evtchn_port_t port;
+        unsigned long pending;
+        int bound;
+};
 
 extern struct file {
     enum fd_type type;
@@ -158,13 +164,7 @@ extern struct file {
 	    off_t offset;
 	} file;
 	struct {
-            /* To each event channel FD is associated a series of ports which
-             * wakes select for this FD. */
-            struct {
-                evtchn_port_t port;
-                unsigned long pending;
-                int bound;
-            } ports[MAX_EVTCHN_PORTS];
+	    struct minios_list_head ports;
 	} evtchn;
 	struct gntmap gntmap;
 	struct {
diff --git a/tools/libxc/xc_minios.c b/tools/libxc/xc_minios.c
index 8bbfd18..29cce63 100644
--- a/tools/libxc/xc_minios.c
+++ b/tools/libxc/xc_minios.c
@@ -210,15 +210,34 @@ static struct xc_osdep_ops minios_privcmd_ops = {
     },
 };
 
+
+/* XXX Note: This is not threadsafe */
+static struct evtchn_port_info* port_alloc(int fd) {
+    struct evtchn_port_info *port_info;
+    port_info = malloc(sizeof(struct evtchn_port_info));
+    if (port_info == NULL)
+        return NULL;
+    port_info->pending = 0;
+    port_info->port = -1;
+    port_info->bound = 0;
+
+    minios_list_add(&port_info->list, &files[fd].evtchn.ports);
+    return port_info;
+}
+
+static void port_dealloc(struct evtchn_port_info *port_info) {
+    if (port_info->bound)
+        unbind_evtchn(port_info->port);
+    minios_list_del(&port_info->list);
+    free(port_info);
+}
+
 static xc_osdep_handle minios_evtchn_open(xc_evtchn *xce)
 {
-    int fd = alloc_fd(FTYPE_EVTCHN), i;
+    int fd = alloc_fd(FTYPE_EVTCHN);
     if ( fd == -1 )
         return XC_OSDEP_OPEN_ERROR;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++) {
-	files[fd].evtchn.ports[i].port = -1;
-        files[fd].evtchn.ports[i].bound = 0;
-    }
+    MINIOS_INIT_LIST_HEAD(&files[fd].evtchn.ports);
     printf("evtchn_open() -> %d\n", fd);
     return (xc_osdep_handle)fd;
 }
@@ -231,10 +250,10 @@ static int minios_evtchn_close(xc_evtchn *xce, xc_osdep_handle h)
 
 void minios_evtchn_close_fd(int fd)
 {
-    int i;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++)
-        if (files[fd].evtchn.ports[i].bound)
-            unbind_evtchn(files[fd].evtchn.ports[i].port);
+    struct evtchn_port_info *port_info, *tmp;
+    minios_list_for_each_entry_safe(port_info, tmp, &files[fd].evtchn.ports, list)
+        port_dealloc(port_info);
+
     files[fd].type = FTYPE_NONE;
 }
 
@@ -256,35 +275,21 @@ static int minios_evtchn_notify(xc_evtchn *xce, xc_osdep_handle h, evtchn_port_t
     return ret;
 }
 
-/* XXX Note: This is not threadsafe */
-static int port_alloc(int fd) {
-    int i;
-    for (i= 0; i < MAX_EVTCHN_PORTS; i++)
-	if (files[fd].evtchn.ports[i].port == -1)
-	    break;
-    if (i == MAX_EVTCHN_PORTS) {
-	printf("Too many ports in xc handle\n");
-	errno = EMFILE;
-	return -1;
-    }
-    files[fd].evtchn.ports[i].pending = 0;
-    return i;
-}
-
 static void evtchn_handler(evtchn_port_t port, struct pt_regs *regs, void *data)
 {
     int fd = (int)(intptr_t)data;
-    int i;
+    struct evtchn_port_info *port_info;
     assert(files[fd].type == FTYPE_EVTCHN);
     mask_evtchn(port);
-    for (i= 0; i < MAX_EVTCHN_PORTS; i++)
-	if (files[fd].evtchn.ports[i].port == port)
-	    break;
-    if (i == MAX_EVTCHN_PORTS) {
-	printk("Unknown port for handle %d\n", fd);
-	return;
+    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
+        if (port_info->port == port)
+            goto found;
     }
-    files[fd].evtchn.ports[i].pending = 1;
+    printk("Unknown port for handle %d\n", fd);
+    return;
+
+ found:
+    port_info->pending = 1;
     files[fd].read = 1;
     wake_up(&event_queue);
 }
@@ -292,12 +297,13 @@ static void evtchn_handler(evtchn_port_t port, struct pt_regs *regs, void *data)
 static evtchn_port_or_error_t minios_evtchn_bind_unbound_port(xc_evtchn *xce, xc_osdep_handle h, int domid)
 {
     int fd = (int)h;
-    int ret, i;
+    struct evtchn_port_info *port_info;
+    int ret;
     evtchn_port_t port;
 
     assert(get_current() == main_thread);
-    i = port_alloc(fd);
-    if (i == -1)
+    port_info = port_alloc(fd);
+    if (port_info == NULL)
 	return -1;
 
     printf("xc_evtchn_bind_unbound_port(%d)", domid);
@@ -305,11 +311,12 @@ static evtchn_port_or_error_t minios_evtchn_bind_unbound_port(xc_evtchn *xce, xc
     printf(" = %d\n", ret);
 
     if (ret < 0) {
+	port_dealloc(port_info);
 	errno = -ret;
 	return -1;
     }
-    files[fd].evtchn.ports[i].bound = 1;
-    files[fd].evtchn.ports[i].port = port;
+    port_info->bound = 1;
+    port_info->port = port;
     unmask_evtchn(port);
     return port;
 }
@@ -318,12 +325,13 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
     evtchn_port_t remote_port)
 {
     int fd = (int)h;
+    struct evtchn_port_info *port_info;
     evtchn_port_t local_port;
-    int ret, i;
+    int ret;
 
     assert(get_current() == main_thread);
-    i = port_alloc(fd);
-    if (i == -1)
+    port_info = port_alloc(fd);
+    if (port_info == NULL)
 	return -1;
 
     printf("xc_evtchn_bind_interdomain(%d, %"PRId32")", domid, remote_port);
@@ -331,11 +339,12 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
     printf(" = %d\n", ret);
 
     if (ret < 0) {
+	port_dealloc(port_info);
 	errno = -ret;
 	return -1;
     }
-    files[fd].evtchn.ports[i].bound = 1;
-    files[fd].evtchn.ports[i].port = local_port;
+    port_info->bound = 1;
+    port_info->port = local_port;
     unmask_evtchn(local_port);
     return local_port;
 }
@@ -343,42 +352,40 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
 static int minios_evtchn_unbind(xc_evtchn *xce, xc_osdep_handle h, evtchn_port_t port)
 {
     int fd = (int)h;
-    int i;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++)
-	if (files[fd].evtchn.ports[i].port == port) {
-	    files[fd].evtchn.ports[i].port = -1;
-	    break;
-	}
-    if (i == MAX_EVTCHN_PORTS) {
-	printf("Warning: couldn't find port %"PRId32" for xc handle %x\n", port, fd);
-	errno = -EINVAL;
-	return -1;
+    struct evtchn_port_info *port_info;
+
+    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
+        if (port_info->port == port) {
+            port_dealloc(port_info);
+            return 0;
+        }
     }
-    files[fd].evtchn.ports[i].bound = 0;
-    unbind_evtchn(port);
-    return 0;
+    printf("Warning: couldn't find port %"PRId32" for xc handle %x\n", port, fd);
+    errno = -EINVAL;
+    return -1;
 }
 
 static evtchn_port_or_error_t minios_evtchn_bind_virq(xc_evtchn *xce, xc_osdep_handle h, unsigned int virq)
 {
     int fd = (int)h;
+    struct evtchn_port_info *port_info;
     evtchn_port_t port;
-    int i;
 
     assert(get_current() == main_thread);
-    i = port_alloc(fd);
-    if (i == -1)
+    port_info = port_alloc(fd);
+    if (port_info == NULL)
 	return -1;
 
     printf("xc_evtchn_bind_virq(%d)", virq);
     port = bind_virq(virq, evtchn_handler, (void*)(intptr_t)fd);
 
     if (port < 0) {
+	port_dealloc(port_info);
 	errno = -port;
 	return -1;
     }
-    files[fd].evtchn.ports[i].bound = 1;
-    files[fd].evtchn.ports[i].port = port;
+    port_info->bound = 1;
+    port_info->port = port;
     unmask_evtchn(port);
     return port;
 }
@@ -386,18 +393,18 @@ static evtchn_port_or_error_t minios_evtchn_bind_virq(xc_evtchn *xce, xc_osdep_h
 static evtchn_port_or_error_t minios_evtchn_pending(xc_evtchn *xce, xc_osdep_handle h)
 {
     int fd = (int)h;
-    int i;
+    struct evtchn_port_info *port_info;
     unsigned long flags;
     evtchn_port_t ret = -1;
 
     local_irq_save(flags);
     files[fd].read = 0;
-    for (i = 0; i < MAX_EVTCHN_PORTS; i++) {
-        evtchn_port_t port = files[fd].evtchn.ports[i].port;
-        if (port != -1 && files[fd].evtchn.ports[i].pending) {
+
+    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
+        if (port_info->port != -1 && port_info->pending) {
             if (ret == -1) {
-                ret = port;
-                files[fd].evtchn.ports[i].pending = 0;
+                ret = port_info->port;
+                port_info->pending = 0;
             } else {
                 files[fd].read = 1;
                 break;
-- 
1.7.7.5

Re: [PATCH 09/18] mini-os: remove per-fd evtchn limit

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> 
> Changes the minios evtchn implementation to use a list instead of an array.
> This allows it to grow as necessary to support any number of ports.
> Unfortunately, it's still limited by NR_EVS in events.c.

NR_EVS is 1024 which, IIRC, is the actual ABI limit for a 32 bit domain.
In principal we could make this 4096 for a 64 bit stub domain but I
guess we only need 1 evtchn per domain plus perhaps a couple of
incidental global ones (e.g. console, VIRQ_mumble) and ~1000+ domains is
more than enough for any current system.

> Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  extras/mini-os/include/lib.h |   16 +++---
>  tools/libxc/xc_minios.c      |  139 ++++++++++++++++++++++--------------------
>  2 files changed, 81 insertions(+), 74 deletions(-)
> 
> diff --git a/extras/mini-os/include/lib.h b/extras/mini-os/include/lib.h
> index bd3eeaf..12070c3 100644
> --- a/extras/mini-os/include/lib.h
> +++ b/extras/mini-os/include/lib.h
> @@ -53,6 +53,7 @@
>  #include <xen/xen.h>
>  #include <xen/event_channel.h>
>  #include "gntmap.h"
> +#include "list.h"
>  
>  #ifdef HAVE_LIBC
>  #include <stdio.h>
> @@ -143,7 +144,12 @@ enum fd_type {
>      FTYPE_SAVEFILE,
>  };
>  
> -#define MAX_EVTCHN_PORTS 16
> +struct evtchn_port_info {
> +        struct minios_list_head list;
> +        evtchn_port_t port;
> +        unsigned long pending;
> +        int bound;
> +};
>  
>  extern struct file {
>      enum fd_type type;
> @@ -158,13 +164,7 @@ extern struct file {
>  	    off_t offset;
>  	} file;
>  	struct {
> -            /* To each event channel FD is associated a series of ports which
> -             * wakes select for this FD. */
> -            struct {
> -                evtchn_port_t port;
> -                unsigned long pending;
> -                int bound;
> -            } ports[MAX_EVTCHN_PORTS];
> +	    struct minios_list_head ports;
>  	} evtchn;
>  	struct gntmap gntmap;
>  	struct {
> diff --git a/tools/libxc/xc_minios.c b/tools/libxc/xc_minios.c
> index 8bbfd18..29cce63 100644
> --- a/tools/libxc/xc_minios.c
> +++ b/tools/libxc/xc_minios.c
> @@ -210,15 +210,34 @@ static struct xc_osdep_ops minios_privcmd_ops = {
>      },
>  };
>  
> +
> +/* XXX Note: This is not threadsafe */
> +static struct evtchn_port_info* port_alloc(int fd) {
> +    struct evtchn_port_info *port_info;
> +    port_info = malloc(sizeof(struct evtchn_port_info));
> +    if (port_info == NULL)
> +        return NULL;
> +    port_info->pending = 0;
> +    port_info->port = -1;
> +    port_info->bound = 0;
> +
> +    minios_list_add(&port_info->list, &files[fd].evtchn.ports);
> +    return port_info;
> +}
> +
> +static void port_dealloc(struct evtchn_port_info *port_info) {
> +    if (port_info->bound)
> +        unbind_evtchn(port_info->port);
> +    minios_list_del(&port_info->list);
> +    free(port_info);
> +}
> +
>  static xc_osdep_handle minios_evtchn_open(xc_evtchn *xce)
>  {
> -    int fd = alloc_fd(FTYPE_EVTCHN), i;
> +    int fd = alloc_fd(FTYPE_EVTCHN);
>      if ( fd == -1 )
>          return XC_OSDEP_OPEN_ERROR;
> -    for (i = 0; i < MAX_EVTCHN_PORTS; i++) {
> -	files[fd].evtchn.ports[i].port = -1;
> -        files[fd].evtchn.ports[i].bound = 0;
> -    }
> +    MINIOS_INIT_LIST_HEAD(&files[fd].evtchn.ports);
>      printf("evtchn_open() -> %d\n", fd);
>      return (xc_osdep_handle)fd;
>  }
> @@ -231,10 +250,10 @@ static int minios_evtchn_close(xc_evtchn *xce, xc_osdep_handle h)
>  
>  void minios_evtchn_close_fd(int fd)
>  {
> -    int i;
> -    for (i = 0; i < MAX_EVTCHN_PORTS; i++)
> -        if (files[fd].evtchn.ports[i].bound)
> -            unbind_evtchn(files[fd].evtchn.ports[i].port);
> +    struct evtchn_port_info *port_info, *tmp;
> +    minios_list_for_each_entry_safe(port_info, tmp, &files[fd].evtchn.ports, list)
> +        port_dealloc(port_info);
> +
>      files[fd].type = FTYPE_NONE;
>  }
>  
> @@ -256,35 +275,21 @@ static int minios_evtchn_notify(xc_evtchn *xce, xc_osdep_handle h, evtchn_port_t
>      return ret;
>  }
>  
> -/* XXX Note: This is not threadsafe */
> -static int port_alloc(int fd) {
> -    int i;
> -    for (i= 0; i < MAX_EVTCHN_PORTS; i++)
> -	if (files[fd].evtchn.ports[i].port == -1)
> -	    break;
> -    if (i == MAX_EVTCHN_PORTS) {
> -	printf("Too many ports in xc handle\n");
> -	errno = EMFILE;
> -	return -1;
> -    }
> -    files[fd].evtchn.ports[i].pending = 0;
> -    return i;
> -}
> -
>  static void evtchn_handler(evtchn_port_t port, struct pt_regs *regs, void *data)
>  {
>      int fd = (int)(intptr_t)data;
> -    int i;
> +    struct evtchn_port_info *port_info;
>      assert(files[fd].type == FTYPE_EVTCHN);
>      mask_evtchn(port);
> -    for (i= 0; i < MAX_EVTCHN_PORTS; i++)
> -	if (files[fd].evtchn.ports[i].port == port)
> -	    break;
> -    if (i == MAX_EVTCHN_PORTS) {
> -	printk("Unknown port for handle %d\n", fd);
> -	return;
> +    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
> +        if (port_info->port == port)
> +            goto found;
>      }
> -    files[fd].evtchn.ports[i].pending = 1;
> +    printk("Unknown port for handle %d\n", fd);
> +    return;
> +
> + found:
> +    port_info->pending = 1;
>      files[fd].read = 1;
>      wake_up(&event_queue);
>  }
> @@ -292,12 +297,13 @@ static void evtchn_handler(evtchn_port_t port, struct pt_regs *regs, void *data)
>  static evtchn_port_or_error_t minios_evtchn_bind_unbound_port(xc_evtchn *xce, xc_osdep_handle h, int domid)
>  {
>      int fd = (int)h;
> -    int ret, i;
> +    struct evtchn_port_info *port_info;
> +    int ret;
>      evtchn_port_t port;
>  
>      assert(get_current() == main_thread);
> -    i = port_alloc(fd);
> -    if (i == -1)
> +    port_info = port_alloc(fd);
> +    if (port_info == NULL)
>  	return -1;
>  
>      printf("xc_evtchn_bind_unbound_port(%d)", domid);
> @@ -305,11 +311,12 @@ static evtchn_port_or_error_t minios_evtchn_bind_unbound_port(xc_evtchn *xce, xc
>      printf(" = %d\n", ret);
>  
>      if (ret < 0) {
> +	port_dealloc(port_info);
>  	errno = -ret;
>  	return -1;
>      }
> -    files[fd].evtchn.ports[i].bound = 1;
> -    files[fd].evtchn.ports[i].port = port;
> +    port_info->bound = 1;
> +    port_info->port = port;
>      unmask_evtchn(port);
>      return port;
>  }
> @@ -318,12 +325,13 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
>      evtchn_port_t remote_port)
>  {
>      int fd = (int)h;
> +    struct evtchn_port_info *port_info;
>      evtchn_port_t local_port;
> -    int ret, i;
> +    int ret;
>  
>      assert(get_current() == main_thread);
> -    i = port_alloc(fd);
> -    if (i == -1)
> +    port_info = port_alloc(fd);
> +    if (port_info == NULL)
>  	return -1;
>  
>      printf("xc_evtchn_bind_interdomain(%d, %"PRId32")", domid, remote_port);
> @@ -331,11 +339,12 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
>      printf(" = %d\n", ret);
>  
>      if (ret < 0) {
> +	port_dealloc(port_info);
>  	errno = -ret;
>  	return -1;
>      }
> -    files[fd].evtchn.ports[i].bound = 1;
> -    files[fd].evtchn.ports[i].port = local_port;
> +    port_info->bound = 1;
> +    port_info->port = local_port;
>      unmask_evtchn(local_port);
>      return local_port;
>  }
> @@ -343,42 +352,40 @@ static evtchn_port_or_error_t minios_evtchn_bind_interdomain(xc_evtchn *xce, xc_
>  static int minios_evtchn_unbind(xc_evtchn *xce, xc_osdep_handle h, evtchn_port_t port)
>  {
>      int fd = (int)h;
> -    int i;
> -    for (i = 0; i < MAX_EVTCHN_PORTS; i++)
> -	if (files[fd].evtchn.ports[i].port == port) {
> -	    files[fd].evtchn.ports[i].port = -1;
> -	    break;
> -	}
> -    if (i == MAX_EVTCHN_PORTS) {
> -	printf("Warning: couldn't find port %"PRId32" for xc handle %x\n", port, fd);
> -	errno = -EINVAL;
> -	return -1;
> +    struct evtchn_port_info *port_info;
> +
> +    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
> +        if (port_info->port == port) {
> +            port_dealloc(port_info);
> +            return 0;
> +        }
>      }
> -    files[fd].evtchn.ports[i].bound = 0;
> -    unbind_evtchn(port);
> -    return 0;
> +    printf("Warning: couldn't find port %"PRId32" for xc handle %x\n", port, fd);
> +    errno = -EINVAL;
> +    return -1;
>  }
>  
>  static evtchn_port_or_error_t minios_evtchn_bind_virq(xc_evtchn *xce, xc_osdep_handle h, unsigned int virq)
>  {
>      int fd = (int)h;
> +    struct evtchn_port_info *port_info;
>      evtchn_port_t port;
> -    int i;
>  
>      assert(get_current() == main_thread);
> -    i = port_alloc(fd);
> -    if (i == -1)
> +    port_info = port_alloc(fd);
> +    if (port_info == NULL)
>  	return -1;
>  
>      printf("xc_evtchn_bind_virq(%d)", virq);
>      port = bind_virq(virq, evtchn_handler, (void*)(intptr_t)fd);
>  
>      if (port < 0) {
> +	port_dealloc(port_info);
>  	errno = -port;
>  	return -1;
>      }
> -    files[fd].evtchn.ports[i].bound = 1;
> -    files[fd].evtchn.ports[i].port = port;
> +    port_info->bound = 1;
> +    port_info->port = port;
>      unmask_evtchn(port);
>      return port;
>  }
> @@ -386,18 +393,18 @@ static evtchn_port_or_error_t minios_evtchn_bind_virq(xc_evtchn *xce, xc_osdep_h
>  static evtchn_port_or_error_t minios_evtchn_pending(xc_evtchn *xce, xc_osdep_handle h)
>  {
>      int fd = (int)h;
> -    int i;
> +    struct evtchn_port_info *port_info;
>      unsigned long flags;
>      evtchn_port_t ret = -1;
>  
>      local_irq_save(flags);
>      files[fd].read = 0;
> -    for (i = 0; i < MAX_EVTCHN_PORTS; i++) {
> -        evtchn_port_t port = files[fd].evtchn.ports[i].port;
> -        if (port != -1 && files[fd].evtchn.ports[i].pending) {
> +
> +    minios_list_for_each_entry(port_info, &files[fd].evtchn.ports, list) {
> +        if (port_info->port != -1 && port_info->pending) {
>              if (ret == -1) {
> -                ret = port;
> -                files[fd].evtchn.ports[i].pending = 0;
> +                ret = port_info->port;
> +                port_info->pending = 0;
>              } else {
>                  files[fd].read = 1;
>                  break;

[PATCH 10/18] xenstored: use grant references instead of map_foreign_range

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

make xenstored use grantref rather than map_foreign_range (which can
only be used by privileged domains)

This patch modifies the xenstore daemon to use xc_gnttab_map_grant_ref
instead of xc_map_foreign_range where available.

Previous versions of this patch have been sent to xen-devel. See
http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00610.html
http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01492.html

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_domain.c |   45 ++++++++++++++++++++++++++++++++-----
 1 files changed, 39 insertions(+), 6 deletions(-)

diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 443af82..0b8353b 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -32,8 +32,10 @@
 #include "xenstored_watch.h"
 
 #include <xenctrl.h>
+#include <xen/grant_table.h>
 
 static xc_interface **xc_handle;
+static xc_gnttab **xcg_handle;
 static evtchn_port_t virq_port;
 
 xc_evtchn *xce_handle = NULL;
@@ -174,8 +176,12 @@ static int destroy_domain(void *_domain)
 			eprintf("> Unbinding port %i failed!\n", domain->port);
 	}
 
-	if (domain->interface)
-		munmap(domain->interface, getpagesize());
+	if (domain->interface) {
+		if (*xcg_handle >= 0 && domain->domid != 0)
+			xc_gnttab_munmap(*xcg_handle, domain->interface, 1);
+		else
+			munmap(domain->interface, getpagesize());
+	}
 
 	fire_watches(NULL, "@releaseDomain", false);
 
@@ -344,9 +350,17 @@ void do_introduce(struct connection *conn, struct buffered_data *in)
 	domain = find_domain_by_domid(domid);
 
 	if (domain == NULL) {
-		interface = xc_map_foreign_range(
-			*xc_handle, domid,
-			getpagesize(), PROT_READ|PROT_WRITE, mfn);
+		if (*xcg_handle >= 0) {
+			/* this is the preferred method */
+			interface = xc_gnttab_map_grant_ref(
+				*xcg_handle, domid,
+				GNTTAB_RESERVED_XENSTORE,
+				PROT_READ|PROT_WRITE);
+		} else {
+			interface = xc_map_foreign_range(
+				*xc_handle, domid,
+				getpagesize(), PROT_READ|PROT_WRITE, mfn);
+		}
 		if (!interface) {
 			send_error(conn, errno);
 			return;
@@ -354,7 +368,10 @@ void do_introduce(struct connection *conn, struct buffered_data *in)
 		/* Hang domain off "in" until we're finished. */
 		domain = new_domain(in, domid, port);
 		if (!domain) {
-			munmap(interface, getpagesize());
+			if (*xcg_handle >= 0)
+				xc_gnttab_munmap(*xcg_handle, interface, 1);
+			else
+				munmap(interface, getpagesize());
 			send_error(conn, errno);
 			return;
 		}
@@ -552,6 +569,12 @@ static int close_xc_handle(void *_handle)
 	return 0;
 }
 
+static int close_xcg_handle(void *_handle)
+{
+	xc_gnttab_close(*(xc_gnttab **)_handle);
+	return 0;
+}
+
 /* Returns the implicit path of a connection (only domains have this) */
 const char *get_implicit_path(const struct connection *conn)
 {
@@ -603,6 +626,16 @@ void domain_init(void)
 
 	talloc_set_destructor(xc_handle, close_xc_handle);
 
+	xcg_handle = talloc(talloc_autofree_context(), xc_gnttab*);
+	if (!xcg_handle)
+		barf_perror("Failed to allocate domain gnttab handle");
+
+	*xcg_handle = xc_gnttab_open(NULL, 0);
+	if (*xcg_handle < 0)
+		xprintf("WARNING: Failed to open connection to gnttab\n");
+	else
+		talloc_set_destructor(xcg_handle, close_xcg_handle);
+
 	xce_handle = xc_evtchn_open(NULL, 0);
 
 	if (xce_handle == NULL)
-- 
1.7.7.5

Re: [PATCH 10/18] xenstored: use grant references instead of map_foreign_range

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> 
> make xenstored use grantref rather than map_foreign_range (which can
> only be used by privileged domains)
> 
> This patch modifies the xenstore daemon to use xc_gnttab_map_grant_ref
> instead of xc_map_foreign_range where available.
> 
> Previous versions of this patch have been sent to xen-devel. See
> http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00610.html
> http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01492.html
> 
> Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  tools/xenstore/xenstored_domain.c |   45 ++++++++++++++++++++++++++++++++-----
>  1 files changed, 39 insertions(+), 6 deletions(-)
> 
> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
> index 443af82..0b8353b 100644
> --- a/tools/xenstore/xenstored_domain.c
> +++ b/tools/xenstore/xenstored_domain.c
> @@ -32,8 +32,10 @@
>  #include "xenstored_watch.h"
>  
>  #include <xenctrl.h>
> +#include <xen/grant_table.h>
>  
>  static xc_interface **xc_handle;
> +static xc_gnttab **xcg_handle;
>  static evtchn_port_t virq_port;
>  
>  xc_evtchn *xce_handle = NULL;
> @@ -174,8 +176,12 @@ static int destroy_domain(void *_domain)
>  			eprintf("> Unbinding port %i failed!\n", domain->port);
>  	}
>  
> -	if (domain->interface)
> -		munmap(domain->interface, getpagesize());
> +	if (domain->interface) {
> +		if (*xcg_handle >= 0 && domain->domid != 0)

Why the special case for domid 0 here? There seems to be no equivalent
for the map case, including the one you add in patch 15/18.

I think the map and unmap logic could usefully be made into helper
functions.

Ian.

Re: [PATCH 10/18] xenstored: use grant references instead of map_foreign_range

[Daniel De Graaf]

On 01/18/2012 06:15 AM, Ian Campbell wrote:
> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
>> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
>>
>> make xenstored use grantref rather than map_foreign_range (which can
>> only be used by privileged domains)
>>
>> This patch modifies the xenstore daemon to use xc_gnttab_map_grant_ref
>> instead of xc_map_foreign_range where available.
>>
>> Previous versions of this patch have been sent to xen-devel. See
>> http://lists.xensource.com/archives/html/xen-devel/2008-07/msg00610.html
>> http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01492.html
>>
>> Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
>> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>> ---
>>  tools/xenstore/xenstored_domain.c |   45 ++++++++++++++++++++++++++++++++-----
>>  1 files changed, 39 insertions(+), 6 deletions(-)
>>
>> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
>> index 443af82..0b8353b 100644
>> --- a/tools/xenstore/xenstored_domain.c
>> +++ b/tools/xenstore/xenstored_domain.c
>> @@ -32,8 +32,10 @@
>>  #include "xenstored_watch.h"
>>  
>>  #include <xenctrl.h>
>> +#include <xen/grant_table.h>
>>  
>>  static xc_interface **xc_handle;
>> +static xc_gnttab **xcg_handle;
>>  static evtchn_port_t virq_port;
>>  
>>  xc_evtchn *xce_handle = NULL;
>> @@ -174,8 +176,12 @@ static int destroy_domain(void *_domain)
>>  			eprintf("> Unbinding port %i failed!\n", domain->port);
>>  	}
>>  
>> -	if (domain->interface)
>> -		munmap(domain->interface, getpagesize());
>> +	if (domain->interface) {
>> +		if (*xcg_handle >= 0 && domain->domid != 0)
> 
> Why the special case for domid 0 here? There seems to be no equivalent
> for the map case, including the one you add in patch 15/18.
 
dom0 is mapped by dom0_init()/xenbus_map() and must be unmapped with munmap,
which is not guaranteed to be the same as xc_gnttab_munmap. The map case
doesn't need to handle that.

> I think the map and unmap logic could usefully be made into helper
> functions.
> 
> Ian.

-- 
Daniel De Graaf
National Security Agency

[PATCH 11/18] xenstored: add NO_SOCKETS compilation option

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

option for compiling xenstored without unix sockets to support running on mini-OS

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c |   22 +++++++++++++++++++++-
 tools/xenstore/xs.c             |    2 ++
 tools/xenstore/xs_lib.c         |    4 ++++
 3 files changed, 27 insertions(+), 1 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 9e6c2c7..631bfe4 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -19,9 +19,11 @@
 
 #include <sys/types.h>
 #include <sys/stat.h>
-#include <sys/socket.h>
 #include <sys/select.h>
+#ifndef NO_SOCKETS
+#include <sys/socket.h>
 #include <sys/un.h>
+#endif
 #include <sys/time.h>
 #include <time.h>
 #include <unistd.h>
@@ -320,8 +322,10 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
 	FD_ZERO(inset);
 	FD_ZERO(outset);
 
+#ifndef NO_SOCKETS
 	set_fd(sock,               inset, &max);
 	set_fd(ro_sock,            inset, &max);
+#endif
 	set_fd(reopen_log_pipe[0], inset, &max);
 
 	if (xce_handle != NULL)
@@ -343,12 +347,14 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
 	return max;
 }
 
+#ifndef NO_SOCKETS
 static int destroy_fd(void *_fd)
 {
 	int *fd = _fd;
 	close(*fd);
 	return 0;
 }
+#endif
 
 /* Is child a subnode of parent, or equal? */
 bool is_child(const char *child, const char *parent)
@@ -1352,6 +1358,7 @@ struct connection *new_connection(connwritefn_t *write, connreadfn_t *read)
 	return new;
 }
 
+#ifndef NO_SOCKETS
 static int writefd(struct connection *conn, const void *data, unsigned int len)
 {
 	int rc;
@@ -1406,6 +1413,7 @@ static void accept_connection(int sock, bool canwrite)
 	} else
 		close(fd);
 }
+#endif
 
 #define TDB_FLAGS 0
 
@@ -1753,7 +1761,11 @@ extern void dump_conn(struct connection *conn);
 int main(int argc, char *argv[])
 {
 	int opt, *sock, *ro_sock, max;
+#ifdef NO_SOCKETS
+	int minus_one = -1;
+#else
 	struct sockaddr_un addr;
+#endif
 	fd_set inset, outset;
 	bool dofork = true;
 	bool outputpid = false;
@@ -1837,6 +1849,9 @@ int main(int argc, char *argv[])
 	if (!dofork)
 		talloc_enable_leak_report_full();
 
+#ifdef NO_SOCKETS
+	sock = ro_sock = &minus_one;
+#else
 	/* Create sockets for them to listen to. */
 	sock = talloc(talloc_autofree_context(), int);
 	*sock = socket(PF_UNIX, SOCK_STREAM, 0);
@@ -1848,10 +1863,12 @@ int main(int argc, char *argv[])
 		barf_perror("Could not create socket");
 	talloc_set_destructor(sock, destroy_fd);
 	talloc_set_destructor(ro_sock, destroy_fd);
+#endif
 
 	/* Don't kill us with SIGPIPE. */
 	signal(SIGPIPE, SIG_IGN);
 
+#ifndef NO_SOCKETS
 	/* FIXME: Be more sophisticated, don't mug running daemon. */
 	unlink(xs_daemon_socket());
 	unlink(xs_daemon_socket_ro());
@@ -1871,6 +1888,7 @@ int main(int argc, char *argv[])
 	if (listen(*sock, 1) != 0
 	    || listen(*ro_sock, 1) != 0)
 		barf_perror("Could not listen on sockets");
+#endif
 
 	if (pipe(reopen_log_pipe)) {
 		barf_perror("pipe");
@@ -1931,11 +1949,13 @@ int main(int argc, char *argv[])
 			reopen_log();
 		}
 
+#ifndef NO_SOCKETS
 		if (FD_ISSET(*sock, &inset))
 			accept_connection(*sock, true);
 
 		if (FD_ISSET(*ro_sock, &inset))
 			accept_connection(*ro_sock, false);
+#endif
 
 		if (evtchn_fd != -1 && FD_ISSET(evtchn_fd, &inset))
 			handle_event();
diff --git a/tools/xenstore/xs.c b/tools/xenstore/xs.c
index 8e54fe0..119d945 100644
--- a/tools/xenstore/xs.c
+++ b/tools/xenstore/xs.c
@@ -271,10 +271,12 @@ struct xs_handle *xs_open(unsigned long flags)
 {
 	struct xs_handle *xsh = NULL;
 
+#ifndef NO_SOCKETS
 	if (flags & XS_OPEN_READONLY)
 		xsh = get_handle(xs_daemon_socket_ro());
 	else
 		xsh = get_handle(xs_daemon_socket());
+#endif
 
 	if (!xsh && !(flags & XS_OPEN_SOCKETONLY))
 		xsh = get_handle(xs_domain_dev());
diff --git a/tools/xenstore/xs_lib.c b/tools/xenstore/xs_lib.c
index 03a9ee4..af3db6b 100644
--- a/tools/xenstore/xs_lib.c
+++ b/tools/xenstore/xs_lib.c
@@ -39,6 +39,7 @@ const char *xs_daemon_rundir(void)
 	return (s ? s : "/var/run/xenstored");
 }
 
+#ifndef NO_SOCKETS
 static const char *xs_daemon_path(void)
 {
 	static char buf[PATH_MAX];
@@ -50,6 +51,7 @@ static const char *xs_daemon_path(void)
 		return NULL;
 	return buf;
 }
+#endif
 
 const char *xs_daemon_tdb(void)
 {
@@ -58,6 +60,7 @@ const char *xs_daemon_tdb(void)
 	return buf;
 }
 
+#ifndef NO_SOCKETS
 const char *xs_daemon_socket(void)
 {
 	return xs_daemon_path();
@@ -73,6 +76,7 @@ const char *xs_daemon_socket_ro(void)
 		return NULL;
 	return buf;
 }
+#endif
 
 const char *xs_domain_dev(void)
 {
-- 
1.7.7.5

Re: [PATCH 11/18] xenstored: add NO_SOCKETS compilation option

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> 
> option for compiling xenstored without unix sockets to support running on mini-OS
> 
> Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

I guess this about as good as it gets, you could avoid some of the
ifdef's with "if (*sock != -1)" type constructions but I don't know that
this is necessarily better.

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/xenstore/xenstored_core.c |   22 +++++++++++++++++++++-
>  tools/xenstore/xs.c             |    2 ++
>  tools/xenstore/xs_lib.c         |    4 ++++
>  3 files changed, 27 insertions(+), 1 deletions(-)
> 
> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
> index 9e6c2c7..631bfe4 100644
> --- a/tools/xenstore/xenstored_core.c
> +++ b/tools/xenstore/xenstored_core.c
> @@ -19,9 +19,11 @@
>  
>  #include <sys/types.h>
>  #include <sys/stat.h>
> -#include <sys/socket.h>
>  #include <sys/select.h>
> +#ifndef NO_SOCKETS
> +#include <sys/socket.h>
>  #include <sys/un.h>
> +#endif
>  #include <sys/time.h>
>  #include <time.h>
>  #include <unistd.h>
> @@ -320,8 +322,10 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
>  	FD_ZERO(inset);
>  	FD_ZERO(outset);
>  
> +#ifndef NO_SOCKETS
>  	set_fd(sock,               inset, &max);
>  	set_fd(ro_sock,            inset, &max);
> +#endif
>  	set_fd(reopen_log_pipe[0], inset, &max);
>  
>  	if (xce_handle != NULL)
> @@ -343,12 +347,14 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
>  	return max;
>  }
>  
> +#ifndef NO_SOCKETS
>  static int destroy_fd(void *_fd)
>  {
>  	int *fd = _fd;
>  	close(*fd);
>  	return 0;
>  }
> +#endif
>  
>  /* Is child a subnode of parent, or equal? */
>  bool is_child(const char *child, const char *parent)
> @@ -1352,6 +1358,7 @@ struct connection *new_connection(connwritefn_t *write, connreadfn_t *read)
>  	return new;
>  }
>  
> +#ifndef NO_SOCKETS
>  static int writefd(struct connection *conn, const void *data, unsigned int len)
>  {
>  	int rc;
> @@ -1406,6 +1413,7 @@ static void accept_connection(int sock, bool canwrite)
>  	} else
>  		close(fd);
>  }
> +#endif
>  
>  #define TDB_FLAGS 0
>  
> @@ -1753,7 +1761,11 @@ extern void dump_conn(struct connection *conn);
>  int main(int argc, char *argv[])
>  {
>  	int opt, *sock, *ro_sock, max;
> +#ifdef NO_SOCKETS
> +	int minus_one = -1;
> +#else
>  	struct sockaddr_un addr;
> +#endif
>  	fd_set inset, outset;
>  	bool dofork = true;
>  	bool outputpid = false;
> @@ -1837,6 +1849,9 @@ int main(int argc, char *argv[])
>  	if (!dofork)
>  		talloc_enable_leak_report_full();
>  
> +#ifdef NO_SOCKETS
> +	sock = ro_sock = &minus_one;
> +#else
>  	/* Create sockets for them to listen to. */
>  	sock = talloc(talloc_autofree_context(), int);
>  	*sock = socket(PF_UNIX, SOCK_STREAM, 0);
> @@ -1848,10 +1863,12 @@ int main(int argc, char *argv[])
>  		barf_perror("Could not create socket");
>  	talloc_set_destructor(sock, destroy_fd);
>  	talloc_set_destructor(ro_sock, destroy_fd);
> +#endif
>  
>  	/* Don't kill us with SIGPIPE. */
>  	signal(SIGPIPE, SIG_IGN);
>  
> +#ifndef NO_SOCKETS
>  	/* FIXME: Be more sophisticated, don't mug running daemon. */
>  	unlink(xs_daemon_socket());
>  	unlink(xs_daemon_socket_ro());
> @@ -1871,6 +1888,7 @@ int main(int argc, char *argv[])
>  	if (listen(*sock, 1) != 0
>  	    || listen(*ro_sock, 1) != 0)
>  		barf_perror("Could not listen on sockets");
> +#endif
>  
>  	if (pipe(reopen_log_pipe)) {
>  		barf_perror("pipe");
> @@ -1931,11 +1949,13 @@ int main(int argc, char *argv[])
>  			reopen_log();
>  		}
>  
> +#ifndef NO_SOCKETS
>  		if (FD_ISSET(*sock, &inset))
>  			accept_connection(*sock, true);
>  
>  		if (FD_ISSET(*ro_sock, &inset))
>  			accept_connection(*ro_sock, false);
> +#endif
>  
>  		if (evtchn_fd != -1 && FD_ISSET(evtchn_fd, &inset))
>  			handle_event();
> diff --git a/tools/xenstore/xs.c b/tools/xenstore/xs.c
> index 8e54fe0..119d945 100644
> --- a/tools/xenstore/xs.c
> +++ b/tools/xenstore/xs.c
> @@ -271,10 +271,12 @@ struct xs_handle *xs_open(unsigned long flags)
>  {
>  	struct xs_handle *xsh = NULL;
>  
> +#ifndef NO_SOCKETS
>  	if (flags & XS_OPEN_READONLY)
>  		xsh = get_handle(xs_daemon_socket_ro());
>  	else
>  		xsh = get_handle(xs_daemon_socket());
> +#endif
>  
>  	if (!xsh && !(flags & XS_OPEN_SOCKETONLY))
>  		xsh = get_handle(xs_domain_dev());
> diff --git a/tools/xenstore/xs_lib.c b/tools/xenstore/xs_lib.c
> index 03a9ee4..af3db6b 100644
> --- a/tools/xenstore/xs_lib.c
> +++ b/tools/xenstore/xs_lib.c
> @@ -39,6 +39,7 @@ const char *xs_daemon_rundir(void)
>  	return (s ? s : "/var/run/xenstored");
>  }
>  
> +#ifndef NO_SOCKETS
>  static const char *xs_daemon_path(void)
>  {
>  	static char buf[PATH_MAX];
> @@ -50,6 +51,7 @@ static const char *xs_daemon_path(void)
>  		return NULL;
>  	return buf;
>  }
> +#endif
>  
>  const char *xs_daemon_tdb(void)
>  {
> @@ -58,6 +60,7 @@ const char *xs_daemon_tdb(void)
>  	return buf;
>  }
>  
> +#ifndef NO_SOCKETS
>  const char *xs_daemon_socket(void)
>  {
>  	return xs_daemon_path();
> @@ -73,6 +76,7 @@ const char *xs_daemon_socket_ro(void)
>  		return NULL;
>  	return buf;
>  }
> +#endif
>  
>  const char *xs_domain_dev(void)
>  {

[PATCH 12/18] xenstored support for in-memory rather than FS based trivial DB (needed to run on mini-OS)

[Daniel De Graaf]

From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>

tdb_copy (a xen modification to tdb?) should honor the TDB_INTERNAL flag
for in-memory databases.

TODO: leaks memory on error case

Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/tdb.c |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/tools/xenstore/tdb.c b/tools/xenstore/tdb.c
index 63205e1..639ce6e 100644
--- a/tools/xenstore/tdb.c
+++ b/tools/xenstore/tdb.c
@@ -2103,6 +2103,42 @@ TDB_CONTEXT *tdb_copy(TDB_CONTEXT *tdb, const char *outfile)
 	int fd, saved_errno;
 	TDB_CONTEXT *copy;
 
+	if (tdb->flags & TDB_INTERNAL) {
+		struct tdb_header *copydb;
+		
+		copy = talloc_zero(outfile, TDB_CONTEXT);
+		if (copy == NULL) {
+			errno = ENOMEM;
+			goto intfail;
+		}
+		memcpy(copy, tdb, sizeof(TDB_CONTEXT));
+
+		if (copy->name || copy->locked || copy->device || copy->inode) {
+			fprintf(stderr, "tdb_copy assumption(s) failed\n");
+			goto intfail;
+		}
+
+		copydb = talloc_zero_size(copy, copy->map_size);
+		if (copydb == NULL) {
+			errno = ENOMEM;
+			goto intfail;
+		}
+		memcpy(copydb, copy->map_ptr, copy->map_size);
+		copy->map_ptr = (char*) copydb;
+
+		if (tdb_brlock(tdb, GLOBAL_LOCK, F_UNLCK, F_SETLKW, 0) == -1)
+			goto intfail;
+
+		copy->next = tdbs;
+		tdbs = copy;
+
+
+		return copy;
+intfail:
+		/* TODO (leaking memory is easier) */
+		return NULL;
+	}
+
 	fd = open(outfile, O_TRUNC|O_CREAT|O_WRONLY, 0640);
 	if (fd < 0)
 		return NULL;
-- 
1.7.7.5

Re: [PATCH 12/18] xenstored support for in-memory rather than FS based trivial DB (needed to run on mini-OS)

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> From: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> 
> tdb_copy (a xen modification to tdb?)

I believe so, the original author of xenstored was also the author of
tdb, IIRC.

>  should honor the TDB_INTERNAL flag
> for in-memory databases.
> 
> TODO: leaks memory on error case

Does this happen on every/any transaction conflict or just if a bad
thing has occurred?

In any case, is fixing it not just a case of calling talloc_free?

FWIW this bit of C xenstored is the main reason I think oxenstored
is/would be a better fit for a stub xenstored.


> Signed-off-by: Diego Ongaro <diego.ongaro@citrix.com>
> Signed-off-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  tools/xenstore/tdb.c |   36 ++++++++++++++++++++++++++++++++++++
>  1 files changed, 36 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/xenstore/tdb.c b/tools/xenstore/tdb.c
> index 63205e1..639ce6e 100644
> --- a/tools/xenstore/tdb.c
> +++ b/tools/xenstore/tdb.c
> @@ -2103,6 +2103,42 @@ TDB_CONTEXT *tdb_copy(TDB_CONTEXT *tdb, const char *outfile)
>  	int fd, saved_errno;
>  	TDB_CONTEXT *copy;
>  
> +	if (tdb->flags & TDB_INTERNAL) {
> +		struct tdb_header *copydb;
> +		
> +		copy = talloc_zero(outfile, TDB_CONTEXT);
> +		if (copy == NULL) {
> +			errno = ENOMEM;
> +			goto intfail;
> +		}
> +		memcpy(copy, tdb, sizeof(TDB_CONTEXT));
> +
> +		if (copy->name || copy->locked || copy->device || copy->inode) {
> +			fprintf(stderr, "tdb_copy assumption(s) failed\n");
> +			goto intfail;
> +		}
> +
> +		copydb = talloc_zero_size(copy, copy->map_size);
> +		if (copydb == NULL) {
> +			errno = ENOMEM;
> +			goto intfail;
> +		}
> +		memcpy(copydb, copy->map_ptr, copy->map_size);
> +		copy->map_ptr = (char*) copydb;
> +
> +		if (tdb_brlock(tdb, GLOBAL_LOCK, F_UNLCK, F_SETLKW, 0) == -1)
> +			goto intfail;
> +
> +		copy->next = tdbs;
> +		tdbs = copy;
> +
> +
> +		return copy;
> +intfail:
> +		/* TODO (leaking memory is easier) */
> +		return NULL;
> +	}
> +
>  	fd = open(outfile, O_TRUNC|O_CREAT|O_WRONLY, 0640);
>  	if (fd < 0)
>  		return NULL;

[PATCH 13/18] xenstored: support running in minios stubdom

[Daniel De Graaf]

A previous versions of this patch has been sent to xen-devel. See
http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01655.html

Originally-by: Diego Ongaro <diego.ongaro@citrix.com>
Originally-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 extras/mini-os/include/list.h          |    6 ++--
 extras/mini-os/main.c                  |    4 +++
 stubdom/Makefile                       |   29 +++++++++++++++++--
 tools/xenstore/Makefile                |    9 +++++-
 tools/xenstore/tdb.c                   |    6 ++--
 tools/xenstore/utils.h                 |    2 +
 tools/xenstore/xenstored_core.c        |   47 ++++++++++++++++++++++++++++++-
 tools/xenstore/xenstored_domain.c      |    7 +++++
 tools/xenstore/xenstored_transaction.c |    2 +
 9 files changed, 100 insertions(+), 12 deletions(-)

diff --git a/extras/mini-os/include/list.h b/extras/mini-os/include/list.h
index a60ae23..4e6a2ac 100644
--- a/extras/mini-os/include/list.h
+++ b/extras/mini-os/include/list.h
@@ -1,5 +1,5 @@
-#ifndef _LINUX_LIST_H
-#define _LINUX_LIST_H
+#ifndef _MINIOS_LIST_H
+#define _MINIOS_LIST_H
 
 /*
  * Simple doubly linked list implementation.
@@ -186,5 +186,5 @@ static __inline__ void minios_list_splice(struct minios_list_head *list, struct
 		n = minios_list_entry(pos->member.next, typeof(*pos), member);	\
 	     &pos->member != (head); 					\
 	     pos = n, n = minios_list_entry(n->member.next, typeof(*n), member))
-#endif /* _LINUX_LIST_H */
+#endif /* _MINIOS_LIST_H */
 
diff --git a/extras/mini-os/main.c b/extras/mini-os/main.c
index b95b889..cd89849 100644
--- a/extras/mini-os/main.c
+++ b/extras/mini-os/main.c
@@ -60,6 +60,7 @@ static void call_main(void *p)
      * crashing. */
     //sleep(1);
 
+#ifndef CONFIG_XENSTORE
 #ifndef CONFIG_GRUB
     sparse((unsigned long) &__app_bss_start, &__app_bss_end - &__app_bss_start);
 #if defined(HAVE_LWIP) && !defined(CONFIG_QEMU)
@@ -67,6 +68,7 @@ static void call_main(void *p)
 #endif
 #endif
     create_thread("pcifront", pcifront_watches, NULL);
+#endif
 
 #ifdef CONFIG_QEMU
     /* Fetch argc, argv from XenStore */
@@ -169,9 +171,11 @@ void _exit(int ret)
     close_all_files();
     __libc_fini_array();
     printk("main returned %d\n", ret);
+#ifndef CONFIG_XENSTORE
 #ifdef HAVE_LWIP
     stop_networking();
 #endif
+#endif
     stop_kernel();
     if (!ret) {
 	/* No problem, just shutdown.  */
diff --git a/stubdom/Makefile b/stubdom/Makefile
index 3705059..e0a90a9 100644
--- a/stubdom/Makefile
+++ b/stubdom/Makefile
@@ -74,14 +74,14 @@ TARGET_CPPFLAGS += -I$(XEN_ROOT)/xen/include
 
 TARGET_LDFLAGS += -nostdlib -L$(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/lib
 
-TARGETS=ioemu c caml grub
+TARGETS=ioemu c caml grub xenstore
 
 CROSS_MAKE := $(MAKE) DESTDIR=
 
 .PHONY: all
 all: build
 ifeq ($(STUBDOM_SUPPORTED),1)
-build: genpath ioemu-stubdom c-stubdom pv-grub
+build: genpath ioemu-stubdom c-stubdom pv-grub xenstore-stubdom
 else
 build: genpath
 endif
@@ -262,6 +262,11 @@ mk-headers-$(XEN_TARGET_ARCH): ioemu/linkfarm.stamp
 	  ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.c . && \
 	  ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.h . && \
 	  ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/Makefile . )
+	mkdir -p xenstore
+	[ -h xenstore/Makefile ] || ( cd xenstore && \
+	  ln -sf $(XEN_ROOT)/tools/xenstore/*.c . && \
+	  ln -sf $(XEN_ROOT)/tools/xenstore/*.h . && \
+	  ln -sf $(XEN_ROOT)/tools/xenstore/Makefile . )
 	$(CROSS_MAKE) -C $(MINI_OS) links
 	touch mk-headers-$(XEN_TARGET_ARCH)
 
@@ -334,6 +339,14 @@ grub: grub-upstream $(CROSS_ROOT)
 	mkdir -p grub-$(XEN_TARGET_ARCH)
 	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(CROSS_MAKE) -C $@ OBJ_DIR=$(CURDIR)/grub-$(XEN_TARGET_ARCH)
 
+##########
+# xenstore
+##########
+
+.PHONY: xenstore
+xenstore: $(CROSS_ROOT)
+	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(CROSS_MAKE) -C $@ LWIPDIR=$(CURDIR)/lwip xenstored.a CONFIG_STUBDOM=y
+
 ########
 # minios
 ########
@@ -355,12 +368,16 @@ c-stubdom: mini-os-$(XEN_TARGET_ARCH)-c lwip-$(XEN_TARGET_ARCH) libxc c
 pv-grub: mini-os-$(XEN_TARGET_ARCH)-grub libxc grub
 	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(CROSS_MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
 
+.PHONY: xenstore-stubdom
+xenstore-stubdom: mini-os-$(XEN_TARGET_ARCH)-xenstore libxc xenstore
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_XENSTORE $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) APP_OBJS=$(CURDIR)/xenstore/xenstored.a
+
 #########
 # install
 #########
 
 ifeq ($(STUBDOM_SUPPORTED),1)
-install: genpath install-readme install-ioemu install-grub
+install: genpath install-readme install-ioemu install-grub install-xenstore
 else
 install: genpath
 endif
@@ -379,6 +396,10 @@ install-grub: pv-grub
 	$(INSTALL_DIR) "$(DESTDIR)$(XENFIRMWAREDIR)"
 	$(INSTALL_DATA) mini-os-$(XEN_TARGET_ARCH)-grub/mini-os.gz "$(DESTDIR)$(XENFIRMWAREDIR)/pv-grub-$(XEN_TARGET_ARCH).gz"
 
+install-xenstore: xenstore-stubdom
+	$(INSTALL_DIR) "$(DESTDIR)/usr/lib/xen/boot"
+	$(INSTALL_PROG) mini-os-$(XEN_TARGET_ARCH)-xenstore/mini-os.gz "$(DESTDIR)/usr/lib/xen/boot/xenstore-stubdom.gz"
+
 #######
 # clean
 #######
@@ -390,12 +411,14 @@ clean:
 	rm -fr mini-os-$(XEN_TARGET_ARCH)-c
 	rm -fr mini-os-$(XEN_TARGET_ARCH)-caml
 	rm -fr mini-os-$(XEN_TARGET_ARCH)-grub
+	rm -fr mini-os-$(XEN_TARGET_ARCH)-xenstore
 	$(CROSS_MAKE) -C caml clean
 	$(CROSS_MAKE) -C c clean
 	rm -fr grub-$(XEN_TARGET_ARCH)
 	rm -f $(STUBDOMPATH)
 	[ ! -d libxc-$(XEN_TARGET_ARCH) ] || $(CROSS_MAKE) -C libxc-$(XEN_TARGET_ARCH) clean
 	-[ ! -d ioemu ] || $(CROSS_MAKE) -C ioemu clean
+	-[ ! -d xenstore ] || $(CROSS_MAKE) -C xenstore clean
 
 # clean the cross-compilation result
 .PHONY: crossclean
diff --git a/tools/xenstore/Makefile b/tools/xenstore/Makefile
index 4facb62..3a061d6 100644
--- a/tools/xenstore/Makefile
+++ b/tools/xenstore/Makefile
@@ -28,6 +28,10 @@ endif
 
 ALL_TARGETS = libxenstore.so libxenstore.a clients xs_tdb_dump xenstored
 
+ifdef CONFIG_STUBDOM
+CFLAGS += -DNO_SOCKETS=1 -DNO_LOCAL_XENBUS=1 -DNO_SYSLOG=1 -DNO_REOPEN_LOG=1
+endif
+
 .PHONY: all
 all: $(ALL_TARGETS)
 
@@ -45,10 +49,13 @@ xenstored_probes.o: xenstored_solaris.o
 
 CFLAGS += -DHAVE_DTRACE=1
 endif
- 
+
 xenstored: $(XENSTORED_OBJS)
 	$(CC) $(LDFLAGS) $^ $(LDLIBS_libxenctrl) $(SOCKET_LIBS) -o $@ $(APPEND_LDFLAGS)
 
+xenstored.a: $(XENSTORED_OBJS)
+	$(AR) cr $@ $^
+
 $(CLIENTS): xenstore
 	ln -f xenstore $@
 
diff --git a/tools/xenstore/tdb.c b/tools/xenstore/tdb.c
index 639ce6e..75ffd2a 100644
--- a/tools/xenstore/tdb.c
+++ b/tools/xenstore/tdb.c
@@ -1334,7 +1334,7 @@ static int tdb_next_lock(TDB_CONTEXT *tdb, struct tdb_traverse_lock *tlock,
 
 		/* Iterate through chain */
 		while( tlock->off) {
-			tdb_off current;
+			tdb_off mycurrent;
 			if (rec_read(tdb, tlock->off, rec) == -1)
 				goto fail;
 
@@ -1352,10 +1352,10 @@ static int tdb_next_lock(TDB_CONTEXT *tdb, struct tdb_traverse_lock *tlock,
 			}
 
 			/* Try to clean dead ones from old traverses */
-			current = tlock->off;
+			mycurrent = tlock->off;
 			tlock->off = rec->next;
 			if (!tdb->read_only && 
-			    do_delete(tdb, current, rec) != 0)
+			    do_delete(tdb, mycurrent, rec) != 0)
 				goto fail;
 		}
 		tdb_unlock(tdb, tlock->hash, F_WRLCK);
diff --git a/tools/xenstore/utils.h b/tools/xenstore/utils.h
index f378343..2effd17 100644
--- a/tools/xenstore/utils.h
+++ b/tools/xenstore/utils.h
@@ -19,7 +19,9 @@ static inline bool strends(const char *a, const char *b)
 	return streq(a + strlen(a) - strlen(b), b);
 }
 
+#ifndef ARRAY_SIZE
 #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
+#endif
 
 void barf(const char *fmt, ...) __attribute__((noreturn));
 void barf_perror(const char *fmt, ...) __attribute__((noreturn));
diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 631bfe4..e51f2ad 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -32,7 +32,9 @@
 #include <stdio.h>
 #include <stdarg.h>
 #include <stdlib.h>
+#ifndef NO_SYSLOG
 #include <syslog.h>
+#endif
 #include <string.h>
 #include <errno.h>
 #include <dirent.h>
@@ -61,13 +63,24 @@ LIST_HEAD(connections);
 static int tracefd = -1;
 static bool recovery = true;
 static bool remove_local = true;
+#ifndef NO_REOPEN_LOG
 static int reopen_log_pipe[2];
+#endif
 static char *tracefile = NULL;
 static TDB_CONTEXT *tdb_ctx;
 
 static void corrupt(struct connection *conn, const char *fmt, ...);
 static void check_store(void);
 
+#ifdef __MINIOS__
+#define lockf(...) (-ENOSYS)
+#endif
+
+#ifdef NO_SYSLOG
+#define openlog(...) ((void) 0)
+#define syslog(...)  ((void) 0)
+#endif
+
 #define log(...)							\
 	do {								\
 		char *s = talloc_asprintf(NULL, __VA_ARGS__);		\
@@ -92,8 +105,10 @@ TDB_CONTEXT *tdb_context(struct connection *conn)
 
 bool replace_tdb(const char *newname, TDB_CONTEXT *newtdb)
 {
+#ifndef __MINIOS__
 	if (rename(newname, xs_daemon_tdb()) != 0)
 		return false;
+#endif
 	tdb_close(tdb_ctx);
 	tdb_ctx = talloc_steal(talloc_autofree_context(), newtdb);
 	return true;
@@ -195,6 +210,7 @@ void trace_destroy(const void *data, const char *type)
 	trace("DESTROY %s %p\n", type, data);
 }
 
+#ifndef NO_REOPEN_LOG
 /**
  * Signal handler for SIGHUP, which requests that the trace log is reopened
  * (in the main loop).  A single byte is written to reopen_log_pipe, to awaken
@@ -222,7 +238,7 @@ static void reopen_log(void)
 			trace("\n***\n");
 	}
 }
-
+#endif
 
 static bool write_messages(struct connection *conn)
 {
@@ -326,7 +342,9 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
 	set_fd(sock,               inset, &max);
 	set_fd(ro_sock,            inset, &max);
 #endif
+#ifndef NO_REOPEN_LOG
 	set_fd(reopen_log_pipe[0], inset, &max);
+#endif
 
 	if (xce_handle != NULL)
 		set_fd(xc_evtchn_fd(xce_handle), inset, &max);
@@ -1415,7 +1433,11 @@ static void accept_connection(int sock, bool canwrite)
 }
 #endif
 
+#ifdef __MINIOS__
+#define TDB_FLAGS TDB_INTERNAL|TDB_NOLOCK
+#else
 #define TDB_FLAGS 0
+#endif
 
 /* We create initial nodes manually. */
 static void manual_node(const char *name, const char *child)
@@ -1440,7 +1462,11 @@ static void setup_structure(void)
 {
 	char *tdbname;
 	tdbname = talloc_strdup(talloc_autofree_context(), xs_daemon_tdb());
+#ifdef __MINIOS__
+	tdb_ctx = NULL;
+#else
 	tdb_ctx = tdb_open(tdbname, 0, TDB_FLAGS, O_RDWR, 0);
+#endif
 
 	if (tdb_ctx) {
 		/* XXX When we make xenstored able to restart, this will have
@@ -1666,6 +1692,7 @@ static void corrupt(struct connection *conn, const char *fmt, ...)
 }
 
 
+#ifndef __MINIOS__
 static void write_pidfile(const char *pidfile)
 {
 	char buf[100];
@@ -1712,7 +1739,7 @@ static void daemonize(void)
 	/* Discard our parent's old-fashioned umask prejudices. */
 	umask(0);
 }
-
+#endif
 
 static void usage(void)
 {
@@ -1767,7 +1794,11 @@ int main(int argc, char *argv[])
 	struct sockaddr_un addr;
 #endif
 	fd_set inset, outset;
+#ifdef __MINIOS__
+	bool dofork = false;
+#else
 	bool dofork = true;
+#endif
 	bool outputpid = false;
 	bool no_domain_init = false;
 	const char *pidfile = NULL;
@@ -1821,8 +1852,11 @@ int main(int argc, char *argv[])
 	if (optind != argc)
 		barf("%s: No arguments desired", argv[0]);
 
+#ifndef NO_REOPEN_LOG
 	reopen_log();
+#endif
 
+#ifndef __MINIOS__
 	/* make sure xenstored directory exists */
 	if (mkdir(xs_daemon_rundir(), 0755)) {
 		if (errno != EEXIST) {
@@ -1844,6 +1878,7 @@ int main(int argc, char *argv[])
 	}
 	if (pidfile)
 		write_pidfile(pidfile);
+#endif
 
 	/* Talloc leak reports go to stderr, which is closed if we fork. */
 	if (!dofork)
@@ -1890,9 +1925,11 @@ int main(int argc, char *argv[])
 		barf_perror("Could not listen on sockets");
 #endif
 
+#ifndef NO_REOPEN_LOG
 	if (pipe(reopen_log_pipe)) {
 		barf_perror("pipe");
 	}
+#endif
 
 	/* Setup the database */
 	setup_structure();
@@ -1921,7 +1958,9 @@ int main(int argc, char *argv[])
 		xprintf = trace;
 	}
 
+#ifndef NO_REOPEN_LOG
 	signal(SIGHUP, trigger_reopen_log);
+#endif
 
 	if (xce_handle != NULL)
 		evtchn_fd = xc_evtchn_fd(xce_handle);
@@ -1929,8 +1968,10 @@ int main(int argc, char *argv[])
 	/* Get ready to listen to the tools. */
 	max = initialize_set(&inset, &outset, *sock, *ro_sock, &timeout);
 
+#ifndef __MINIOS__
 	/* Tell the kernel we're up and running. */
 	xenbus_notify_running();
+#endif
 
 	/* Main loop. */
 	for (;;) {
@@ -1942,12 +1983,14 @@ int main(int argc, char *argv[])
 			barf_perror("Select failed");
 		}
 
+#ifndef NO_REOPEN_LOG
 		if (FD_ISSET(reopen_log_pipe[0], &inset)) {
 			char c;
 			if (read(reopen_log_pipe[0], &c, 1) != 1)
 				barf_perror("read failed");
 			reopen_log();
 		}
+#endif
 
 #ifndef NO_SOCKETS
 		if (FD_ISSET(*sock, &inset))
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 0b8353b..811ae3c 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -588,6 +588,12 @@ void restore_existing_connections(void)
 {
 }
 
+#ifdef __MINIOS__
+static inline int dom0_init(void) 
+{
+	return 0;
+}
+#else
 static int dom0_init(void) 
 { 
 	evtchn_port_t port;
@@ -611,6 +617,7 @@ static int dom0_init(void)
 
 	return 0; 
 }
+#endif
 
 void domain_init(void)
 {
diff --git a/tools/xenstore/xenstored_transaction.c b/tools/xenstore/xenstored_transaction.c
index 380c691..c59acfb 100644
--- a/tools/xenstore/xenstored_transaction.c
+++ b/tools/xenstore/xenstored_transaction.c
@@ -120,7 +120,9 @@ static int destroy_transaction(void *_transaction)
 	trace_destroy(trans, "transaction");
 	if (trans->tdb)
 		tdb_close(trans->tdb);
+#ifndef __MINIOS__
 	unlink(trans->tdb_name);
+#endif
 	return 0;
 }
 
-- 
1.7.7.5

Re: [PATCH 13/18] xenstored: support running in minios stubdom

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> A previous versions of this patch has been sent to xen-devel. See
> http://lists.xensource.com/archives/html/xen-devel/2009-03/msg01655.html
> 
> Originally-by: Diego Ongaro <diego.ongaro@citrix.com>
> Originally-by: Alex Zeffertt <alex.zeffertt@eu.citrix.com>
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

There's a lot of ifdefery here. I don't have any particularly cunning
plans to improve it though :-(

One thing which might help is to provide nop versions of functions
instead of idef'ing both the definition and callsite. e.g. 
 static void write_pidfile(const char *pidfile)
+#ifndef __MINIOS__
     stuff
+#else
+    nothing
+endif

then you avoid another ifdef at the place which calls write_pidfile. I'm
not sure that pattern helps with many of the instances here though.

> ---
>  extras/mini-os/include/list.h          |    6 ++--
>  extras/mini-os/main.c                  |    4 +++
>  stubdom/Makefile                       |   29 +++++++++++++++++--
>  tools/xenstore/Makefile                |    9 +++++-
>  tools/xenstore/tdb.c                   |    6 ++--
>  tools/xenstore/utils.h                 |    2 +
>  tools/xenstore/xenstored_core.c        |   47 ++++++++++++++++++++++++++++++-
>  tools/xenstore/xenstored_domain.c      |    7 +++++
>  tools/xenstore/xenstored_transaction.c |    2 +
>  9 files changed, 100 insertions(+), 12 deletions(-)
> 
> diff --git a/extras/mini-os/include/list.h b/extras/mini-os/include/list.h
> index a60ae23..4e6a2ac 100644
> --- a/extras/mini-os/include/list.h
> +++ b/extras/mini-os/include/list.h
> @@ -1,5 +1,5 @@
> -#ifndef _LINUX_LIST_H
> -#define _LINUX_LIST_H
> +#ifndef _MINIOS_LIST_H
> +#define _MINIOS_LIST_H
> 
>  /*
>   * Simple doubly linked list implementation.
> @@ -186,5 +186,5 @@ static __inline__ void minios_list_splice(struct minios_list_head *list, struct
>                 n = minios_list_entry(pos->member.next, typeof(*pos), member);  \
>              &pos->member != (head);                                    \
>              pos = n, n = minios_list_entry(n->member.next, typeof(*n), member))
> -#endif /* _LINUX_LIST_H */
> +#endif /* _MINIOS_LIST_H */
> 
> diff --git a/extras/mini-os/main.c b/extras/mini-os/main.c
> index b95b889..cd89849 100644
> --- a/extras/mini-os/main.c
> +++ b/extras/mini-os/main.c
> @@ -60,6 +60,7 @@ static void call_main(void *p)
>       * crashing. */
>      //sleep(1);
> 
> +#ifndef CONFIG_XENSTORE
>  #ifndef CONFIG_GRUB
>      sparse((unsigned long) &__app_bss_start, &__app_bss_end - &__app_bss_start);
>  #if defined(HAVE_LWIP) && !defined(CONFIG_QEMU)
> @@ -67,6 +68,7 @@ static void call_main(void *p)
>  #endif
>  #endif
>      create_thread("pcifront", pcifront_watches, NULL);
> +#endif
> 
>  #ifdef CONFIG_QEMU
>      /* Fetch argc, argv from XenStore */
> @@ -169,9 +171,11 @@ void _exit(int ret)
>      close_all_files();
>      __libc_fini_array();
>      printk("main returned %d\n", ret);
> +#ifndef CONFIG_XENSTORE
>  #ifdef HAVE_LWIP
>      stop_networking();
>  #endif
> +#endif
>      stop_kernel();
>      if (!ret) {
>         /* No problem, just shutdown.  */
> diff --git a/stubdom/Makefile b/stubdom/Makefile
> index 3705059..e0a90a9 100644
> --- a/stubdom/Makefile
> +++ b/stubdom/Makefile
> @@ -74,14 +74,14 @@ TARGET_CPPFLAGS += -I$(XEN_ROOT)/xen/include
> 
>  TARGET_LDFLAGS += -nostdlib -L$(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/lib
> 
> -TARGETS=ioemu c caml grub
> +TARGETS=ioemu c caml grub xenstore
> 
>  CROSS_MAKE := $(MAKE) DESTDIR=
> 
>  .PHONY: all
>  all: build
>  ifeq ($(STUBDOM_SUPPORTED),1)
> -build: genpath ioemu-stubdom c-stubdom pv-grub
> +build: genpath ioemu-stubdom c-stubdom pv-grub xenstore-stubdom
>  else
>  build: genpath
>  endif
> @@ -262,6 +262,11 @@ mk-headers-$(XEN_TARGET_ARCH): ioemu/linkfarm.stamp
>           ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.c . && \
>           ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/*.h . && \
>           ln -sf $(XEN_ROOT)/tools/libxc/$(XEN_TARGET_ARCH)/Makefile . )
> +       mkdir -p xenstore
> +       [ -h xenstore/Makefile ] || ( cd xenstore && \
> +         ln -sf $(XEN_ROOT)/tools/xenstore/*.c . && \
> +         ln -sf $(XEN_ROOT)/tools/xenstore/*.h . && \
> +         ln -sf $(XEN_ROOT)/tools/xenstore/Makefile . )
>         $(CROSS_MAKE) -C $(MINI_OS) links
>         touch mk-headers-$(XEN_TARGET_ARCH)
> 
> @@ -334,6 +339,14 @@ grub: grub-upstream $(CROSS_ROOT)
>         mkdir -p grub-$(XEN_TARGET_ARCH)
>         CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(CROSS_MAKE) -C $@ OBJ_DIR=$(CURDIR)/grub-$(XEN_TARGET_ARCH)
> 
> +##########
> +# xenstore
> +##########
> +
> +.PHONY: xenstore
> +xenstore: $(CROSS_ROOT)
> +       CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(CROSS_MAKE) -C $@ LWIPDIR=$(CURDIR)/lwip xenstored.a CONFIG_STUBDOM=y
> +
>  ########
>  # minios
>  ########
> @@ -355,12 +368,16 @@ c-stubdom: mini-os-$(XEN_TARGET_ARCH)-c lwip-$(XEN_TARGET_ARCH) libxc c
>  pv-grub: mini-os-$(XEN_TARGET_ARCH)-grub libxc grub
>         DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(CROSS_MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
> 
> +.PHONY: xenstore-stubdom
> +xenstore-stubdom: mini-os-$(XEN_TARGET_ARCH)-xenstore libxc xenstore
> +       DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_XENSTORE $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< LWIPDIR=$(CURDIR)/lwip-$(XEN_TARGET_ARCH) APP_OBJS=$(CURDIR)/xenstore/xenstored.a
> +
>  #########
>  # install
>  #########
> 
>  ifeq ($(STUBDOM_SUPPORTED),1)
> -install: genpath install-readme install-ioemu install-grub
> +install: genpath install-readme install-ioemu install-grub install-xenstore
>  else
>  install: genpath
>  endif
> @@ -379,6 +396,10 @@ install-grub: pv-grub
>         $(INSTALL_DIR) "$(DESTDIR)$(XENFIRMWAREDIR)"
>         $(INSTALL_DATA) mini-os-$(XEN_TARGET_ARCH)-grub/mini-os.gz "$(DESTDIR)$(XENFIRMWAREDIR)/pv-grub-$(XEN_TARGET_ARCH).gz"
> 
> +install-xenstore: xenstore-stubdom
> +       $(INSTALL_DIR) "$(DESTDIR)/usr/lib/xen/boot"
> +       $(INSTALL_PROG) mini-os-$(XEN_TARGET_ARCH)-xenstore/mini-os.gz "$(DESTDIR)/usr/lib/xen/boot/xenstore-stubdom.gz"
> +
>  #######
>  # clean
>  #######
> @@ -390,12 +411,14 @@ clean:
>         rm -fr mini-os-$(XEN_TARGET_ARCH)-c
>         rm -fr mini-os-$(XEN_TARGET_ARCH)-caml
>         rm -fr mini-os-$(XEN_TARGET_ARCH)-grub
> +       rm -fr mini-os-$(XEN_TARGET_ARCH)-xenstore
>         $(CROSS_MAKE) -C caml clean
>         $(CROSS_MAKE) -C c clean
>         rm -fr grub-$(XEN_TARGET_ARCH)
>         rm -f $(STUBDOMPATH)
>         [ ! -d libxc-$(XEN_TARGET_ARCH) ] || $(CROSS_MAKE) -C libxc-$(XEN_TARGET_ARCH) clean
>         -[ ! -d ioemu ] || $(CROSS_MAKE) -C ioemu clean
> +       -[ ! -d xenstore ] || $(CROSS_MAKE) -C xenstore clean
> 
>  # clean the cross-compilation result
>  .PHONY: crossclean
> diff --git a/tools/xenstore/Makefile b/tools/xenstore/Makefile
> index 4facb62..3a061d6 100644
> --- a/tools/xenstore/Makefile
> +++ b/tools/xenstore/Makefile
> @@ -28,6 +28,10 @@ endif
> 
>  ALL_TARGETS = libxenstore.so libxenstore.a clients xs_tdb_dump xenstored
> 
> +ifdef CONFIG_STUBDOM
> +CFLAGS += -DNO_SOCKETS=1 -DNO_LOCAL_XENBUS=1 -DNO_SYSLOG=1 -DNO_REOPEN_LOG=1
> +endif
> +
>  .PHONY: all
>  all: $(ALL_TARGETS)
> 
> @@ -45,10 +49,13 @@ xenstored_probes.o: xenstored_solaris.o
> 
>  CFLAGS += -DHAVE_DTRACE=1
>  endif
> -
> +
>  xenstored: $(XENSTORED_OBJS)
>         $(CC) $(LDFLAGS) $^ $(LDLIBS_libxenctrl) $(SOCKET_LIBS) -o $@ $(APPEND_LDFLAGS)
> 
> +xenstored.a: $(XENSTORED_OBJS)
> +       $(AR) cr $@ $^
> +
>  $(CLIENTS): xenstore
>         ln -f xenstore $@
> 
> diff --git a/tools/xenstore/tdb.c b/tools/xenstore/tdb.c
> index 639ce6e..75ffd2a 100644
> --- a/tools/xenstore/tdb.c
> +++ b/tools/xenstore/tdb.c
> @@ -1334,7 +1334,7 @@ static int tdb_next_lock(TDB_CONTEXT *tdb, struct tdb_traverse_lock *tlock,
> 
>                 /* Iterate through chain */
>                 while( tlock->off) {
> -                       tdb_off current;
> +                       tdb_off mycurrent;
>                         if (rec_read(tdb, tlock->off, rec) == -1)
>                                 goto fail;
> 
> @@ -1352,10 +1352,10 @@ static int tdb_next_lock(TDB_CONTEXT *tdb, struct tdb_traverse_lock *tlock,
>                         }
> 
>                         /* Try to clean dead ones from old traverses */
> -                       current = tlock->off;
> +                       mycurrent = tlock->off;
>                         tlock->off = rec->next;
>                         if (!tdb->read_only &&
> -                           do_delete(tdb, current, rec) != 0)
> +                           do_delete(tdb, mycurrent, rec) != 0)
>                                 goto fail;
>                 }
>                 tdb_unlock(tdb, tlock->hash, F_WRLCK);
> diff --git a/tools/xenstore/utils.h b/tools/xenstore/utils.h
> index f378343..2effd17 100644
> --- a/tools/xenstore/utils.h
> +++ b/tools/xenstore/utils.h
> @@ -19,7 +19,9 @@ static inline bool strends(const char *a, const char *b)
>         return streq(a + strlen(a) - strlen(b), b);
>  }
> 
> +#ifndef ARRAY_SIZE
>  #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
> +#endif
> 
>  void barf(const char *fmt, ...) __attribute__((noreturn));
>  void barf_perror(const char *fmt, ...) __attribute__((noreturn));
> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
> index 631bfe4..e51f2ad 100644
> --- a/tools/xenstore/xenstored_core.c
> +++ b/tools/xenstore/xenstored_core.c
> @@ -32,7 +32,9 @@
>  #include <stdio.h>
>  #include <stdarg.h>
>  #include <stdlib.h>
> +#ifndef NO_SYSLOG
>  #include <syslog.h>
> +#endif
>  #include <string.h>
>  #include <errno.h>
>  #include <dirent.h>
> @@ -61,13 +63,24 @@ LIST_HEAD(connections);
>  static int tracefd = -1;
>  static bool recovery = true;
>  static bool remove_local = true;
> +#ifndef NO_REOPEN_LOG
>  static int reopen_log_pipe[2];
> +#endif
>  static char *tracefile = NULL;
>  static TDB_CONTEXT *tdb_ctx;
> 
>  static void corrupt(struct connection *conn, const char *fmt, ...);
>  static void check_store(void);
> 
> +#ifdef __MINIOS__
> +#define lockf(...) (-ENOSYS)
> +#endif
> +
> +#ifdef NO_SYSLOG
> +#define openlog(...) ((void) 0)
> +#define syslog(...)  ((void) 0)
> +#endif
> +
>  #define log(...)                                                       \
>         do {                                                            \
>                 char *s = talloc_asprintf(NULL, __VA_ARGS__);           \
> @@ -92,8 +105,10 @@ TDB_CONTEXT *tdb_context(struct connection *conn)
> 
>  bool replace_tdb(const char *newname, TDB_CONTEXT *newtdb)
>  {
> +#ifndef __MINIOS__
>         if (rename(newname, xs_daemon_tdb()) != 0)
>                 return false;
> +#endif
>         tdb_close(tdb_ctx);
>         tdb_ctx = talloc_steal(talloc_autofree_context(), newtdb);
>         return true;
> @@ -195,6 +210,7 @@ void trace_destroy(const void *data, const char *type)
>         trace("DESTROY %s %p\n", type, data);
>  }
> 
> +#ifndef NO_REOPEN_LOG
>  /**
>   * Signal handler for SIGHUP, which requests that the trace log is reopened
>   * (in the main loop).  A single byte is written to reopen_log_pipe, to awaken
> @@ -222,7 +238,7 @@ static void reopen_log(void)
>                         trace("\n***\n");
>         }
>  }
> -
> +#endif
> 
>  static bool write_messages(struct connection *conn)
>  {
> @@ -326,7 +342,9 @@ static int initialize_set(fd_set *inset, fd_set *outset, int sock, int ro_sock,
>         set_fd(sock,               inset, &max);
>         set_fd(ro_sock,            inset, &max);
>  #endif
> +#ifndef NO_REOPEN_LOG
>         set_fd(reopen_log_pipe[0], inset, &max);
> +#endif
> 
>         if (xce_handle != NULL)
>                 set_fd(xc_evtchn_fd(xce_handle), inset, &max);
> @@ -1415,7 +1433,11 @@ static void accept_connection(int sock, bool canwrite)
>  }
>  #endif
> 
> +#ifdef __MINIOS__
> +#define TDB_FLAGS TDB_INTERNAL|TDB_NOLOCK
> +#else
>  #define TDB_FLAGS 0
> +#endif
> 
>  /* We create initial nodes manually. */
>  static void manual_node(const char *name, const char *child)
> @@ -1440,7 +1462,11 @@ static void setup_structure(void)
>  {
>         char *tdbname;
>         tdbname = talloc_strdup(talloc_autofree_context(), xs_daemon_tdb());
> +#ifdef __MINIOS__
> +       tdb_ctx = NULL;
> +#else
>         tdb_ctx = tdb_open(tdbname, 0, TDB_FLAGS, O_RDWR, 0);
> +#endif
> 
>         if (tdb_ctx) {
>                 /* XXX When we make xenstored able to restart, this will have
> @@ -1666,6 +1692,7 @@ static void corrupt(struct connection *conn, const char *fmt, ...)
>  }
> 
> 
> +#ifndef __MINIOS__
>  static void write_pidfile(const char *pidfile)
>  {
>         char buf[100];
> @@ -1712,7 +1739,7 @@ static void daemonize(void)
>         /* Discard our parent's old-fashioned umask prejudices. */
>         umask(0);
>  }
> -
> +#endif
> 
>  static void usage(void)
>  {
> @@ -1767,7 +1794,11 @@ int main(int argc, char *argv[])
>         struct sockaddr_un addr;
>  #endif
>         fd_set inset, outset;
> +#ifdef __MINIOS__
> +       bool dofork = false;
> +#else
>         bool dofork = true;
> +#endif
>         bool outputpid = false;
>         bool no_domain_init = false;
>         const char *pidfile = NULL;
> @@ -1821,8 +1852,11 @@ int main(int argc, char *argv[])
>         if (optind != argc)
>                 barf("%s: No arguments desired", argv[0]);
> 
> +#ifndef NO_REOPEN_LOG
>         reopen_log();
> +#endif
> 
> +#ifndef __MINIOS__
>         /* make sure xenstored directory exists */
>         if (mkdir(xs_daemon_rundir(), 0755)) {
>                 if (errno != EEXIST) {
> @@ -1844,6 +1878,7 @@ int main(int argc, char *argv[])
>         }
>         if (pidfile)
>                 write_pidfile(pidfile);
> +#endif
> 
>         /* Talloc leak reports go to stderr, which is closed if we fork. */
>         if (!dofork)
> @@ -1890,9 +1925,11 @@ int main(int argc, char *argv[])
>                 barf_perror("Could not listen on sockets");
>  #endif
> 
> +#ifndef NO_REOPEN_LOG
>         if (pipe(reopen_log_pipe)) {
>                 barf_perror("pipe");
>         }
> +#endif
> 
>         /* Setup the database */
>         setup_structure();
> @@ -1921,7 +1958,9 @@ int main(int argc, char *argv[])
>                 xprintf = trace;
>         }
> 
> +#ifndef NO_REOPEN_LOG
>         signal(SIGHUP, trigger_reopen_log);
> +#endif
> 
>         if (xce_handle != NULL)
>                 evtchn_fd = xc_evtchn_fd(xce_handle);
> @@ -1929,8 +1968,10 @@ int main(int argc, char *argv[])
>         /* Get ready to listen to the tools. */
>         max = initialize_set(&inset, &outset, *sock, *ro_sock, &timeout);
> 
> +#ifndef __MINIOS__
>         /* Tell the kernel we're up and running. */
>         xenbus_notify_running();
> +#endif
> 
>         /* Main loop. */
>         for (;;) {
> @@ -1942,12 +1983,14 @@ int main(int argc, char *argv[])
>                         barf_perror("Select failed");
>                 }
> 
> +#ifndef NO_REOPEN_LOG
>                 if (FD_ISSET(reopen_log_pipe[0], &inset)) {
>                         char c;
>                         if (read(reopen_log_pipe[0], &c, 1) != 1)
>                                 barf_perror("read failed");
>                         reopen_log();
>                 }
> +#endif
> 
>  #ifndef NO_SOCKETS
>                 if (FD_ISSET(*sock, &inset))
> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
> index 0b8353b..811ae3c 100644
> --- a/tools/xenstore/xenstored_domain.c
> +++ b/tools/xenstore/xenstored_domain.c
> @@ -588,6 +588,12 @@ void restore_existing_connections(void)
>  {
>  }
> 
> +#ifdef __MINIOS__
> +static inline int dom0_init(void)
> +{
> +       return 0;
> +}
> +#else
>  static int dom0_init(void)
>  {
>         evtchn_port_t port;
> @@ -611,6 +617,7 @@ static int dom0_init(void)
> 
>         return 0;
>  }
> +#endif
> 
>  void domain_init(void)
>  {
> diff --git a/tools/xenstore/xenstored_transaction.c b/tools/xenstore/xenstored_transaction.c
> index 380c691..c59acfb 100644
> --- a/tools/xenstore/xenstored_transaction.c
> +++ b/tools/xenstore/xenstored_transaction.c
> @@ -120,7 +120,9 @@ static int destroy_transaction(void *_transaction)
>         trace_destroy(trans, "transaction");
>         if (trans->tdb)
>                 tdb_close(trans->tdb);
> +#ifndef __MINIOS__
>         unlink(trans->tdb_name);
> +#endif
>         return 0;
>  }
> 
> --
> 1.7.7.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 13/18] xenstored: support running in minios stubdom

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 13/18] xenstored: support running in minios stubdom"):
> One thing which might help is to provide nop versions of functions
> instead of idef'ing both the definition and callsite. e.g. 
>  static void write_pidfile(const char *pidfile)
> +#ifndef __MINIOS__
>      stuff
> +#else
> +    nothing
> +endif

I would normally prefer:

> +#ifndef __MINIOS__
>  static void write_pidfile(const char *pidfile)
>      stuff
>  }
> +#else
> +static void write_pidfile(const char *pidfile)
> +}
> +endif

I think this is fairly easy to read; the only hard part is figuring
out which version is being used, which can often be done by putting
the relevant bits in a separate file.

Ian.

Re: [PATCH 13/18] xenstored: support running in minios stubdom

[Ian Campbell]

On Wed, 2012-01-18 at 17:13 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 13/18] xenstored: support running in minios stubdom"):
> > One thing which might help is to provide nop versions of functions
> > instead of idef'ing both the definition and callsite. e.g. 
> >  static void write_pidfile(const char *pidfile)
> > +#ifndef __MINIOS__
> >      stuff
> > +#else
> > +    nothing
> > +endif
> 
> I would normally prefer:
> 
> > +#ifndef __MINIOS__
> >  static void write_pidfile(const char *pidfile)
> >      stuff
> >  }
> > +#else
> > +static void write_pidfile(const char *pidfile)
> > +}
> > +endif

Yes, I'd normally do it this way too, not sure why I wrote the other...

Only real difference is that it prevents the prototype getting out of
sync and bit-rotting the infrequently used case if there is one.

Ian.

> I think this is fairly easy to read; the only hard part is figuring
> out which version is being used, which can often be done by putting
> the relevant bits in a separate file.
> 
> Ian.

Re: [PATCH 13/18] xenstored: support running in minios stubdom

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 13/18] xenstored: support running in minios stubdom"):
> On Wed, 2012-01-18 at 17:13 +0000, Ian Jackson wrote:
> > I would normally prefer:
> > 
> > > +#ifndef __MINIOS__
> > >  static void write_pidfile(const char *pidfile)
> > >      stuff
> > >  }
> > > +#else
> > > +static void write_pidfile(const char *pidfile)
> > > +}
> > > +endif
> 
> Yes, I'd normally do it this way too, not sure why I wrote the other...
> 
> Only real difference is that it prevents the prototype getting out of
> sync and bit-rotting the infrequently used case if there is one.

Better that the prototype gets out of sync and you get a compiler
error, than that the prototype is updated but the little-used
implementations of the body of the function is not adjusted for new
semantics implied by new arguments ...

Ian.

[PATCH 14/18] xenstored: always use xc_gnttab_munmap in stubdom

[Daniel De Graaf]

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_domain.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 811ae3c..aca2149 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -177,10 +177,14 @@ static int destroy_domain(void *_domain)
 	}
 
 	if (domain->interface) {
+#ifdef __MINIOS__
+		xc_gnttab_munmap(*xcg_handle, domain->interface, 1);
+#else
 		if (*xcg_handle >= 0 && domain->domid != 0)
 			xc_gnttab_munmap(*xcg_handle, domain->interface, 1);
 		else
 			munmap(domain->interface, getpagesize());
+#endif
 	}
 
 	fire_watches(NULL, "@releaseDomain", false);
-- 
1.7.7.5

[PATCH 15/18] xenstored: add --event parameter for bootstrapping

[Daniel De Graaf]

When xenstored is run in a minios domain, it needs a bootstrap
connection to dom0 so that additional domain introduce messages can be
sent to it.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c   |    5 +++++
 tools/xenstore/xenstored_core.h   |    1 +
 tools/xenstore/xenstored_domain.c |   13 ++++++++++++-
 3 files changed, 18 insertions(+), 1 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index e51f2ad..4ec63f1 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -1771,6 +1771,7 @@ static struct option options[] = {
 	{ "no-domain-init", 0, NULL, 'D' },
 	{ "entry-nb", 1, NULL, 'E' },
 	{ "pid-file", 1, NULL, 'F' },
+	{ "event", 1, NULL, 'e' },
 	{ "help", 0, NULL, 'H' },
 	{ "no-fork", 0, NULL, 'N' },
 	{ "output-pid", 0, NULL, 'P' },
@@ -1784,6 +1785,7 @@ static struct option options[] = {
 	{ NULL, 0, NULL, 0 } };
 
 extern void dump_conn(struct connection *conn); 
+int dom0_event = 0;
 
 int main(int argc, char *argv[])
 {
@@ -1847,6 +1849,9 @@ int main(int argc, char *argv[])
 		case 'W':
 			quota_nb_watch_per_domain = strtol(optarg, NULL, 10);
 			break;
+		case 'e':
+			dom0_event = strtol(optarg, NULL, 10);
+			break;
 		}
 	}
 	if (optind != argc)
diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
index c487089..d3040ba 100644
--- a/tools/xenstore/xenstored_core.h
+++ b/tools/xenstore/xenstored_core.h
@@ -168,6 +168,7 @@ void trace(const char *fmt, ...);
 void dtrace_io(const struct connection *conn, const struct buffered_data *data, int out);
 
 extern int event_fd;
+extern int dom0_event;
 
 /* Map the kernel's xenstore page. */
 void *xenbus_map(void);
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index aca2149..648eb1d 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -593,8 +593,19 @@ void restore_existing_connections(void)
 }
 
 #ifdef __MINIOS__
-static inline int dom0_init(void) 
+static int dom0_init(void)
 {
+	struct domain *domain;
+	int domid = 0;
+	evtchn_port_t port = dom0_event;
+
+	domain = new_domain(NULL, domid, port);
+	domain->interface = xc_gnttab_map_grant_ref(*xcg_handle, domid,
+			GNTTAB_RESERVED_XENSTORE, PROT_READ|PROT_WRITE);
+	talloc_steal(domain->conn, domain);
+
+	xc_evtchn_notify(xce_handle, domain->port);
+
 	return 0;
 }
 #else
-- 
1.7.7.5

Re: [PATCH 15/18] xenstored: add --event parameter for bootstrapping

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> When xenstored is run in a minios domain, it needs a bootstrap
> connection to dom0 so that additional domain introduce messages can be
> sent to it.
> 
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/xenstore/xenstored_core.c   |    5 +++++
>  tools/xenstore/xenstored_core.h   |    1 +
>  tools/xenstore/xenstored_domain.c |   13 ++++++++++++-
>  3 files changed, 18 insertions(+), 1 deletions(-)
> 
> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
> index e51f2ad..4ec63f1 100644
> --- a/tools/xenstore/xenstored_core.c
> +++ b/tools/xenstore/xenstored_core.c
> @@ -1771,6 +1771,7 @@ static struct option options[] = {
>  	{ "no-domain-init", 0, NULL, 'D' },
>  	{ "entry-nb", 1, NULL, 'E' },
>  	{ "pid-file", 1, NULL, 'F' },
> +	{ "event", 1, NULL, 'e' },
>  	{ "help", 0, NULL, 'H' },
>  	{ "no-fork", 0, NULL, 'N' },
>  	{ "output-pid", 0, NULL, 'P' },
> @@ -1784,6 +1785,7 @@ static struct option options[] = {
>  	{ NULL, 0, NULL, 0 } };
>  
>  extern void dump_conn(struct connection *conn); 
> +int dom0_event = 0;
>  
>  int main(int argc, char *argv[])
>  {
> @@ -1847,6 +1849,9 @@ int main(int argc, char *argv[])
>  		case 'W':
>  			quota_nb_watch_per_domain = strtol(optarg, NULL, 10);
>  			break;
> +		case 'e':
> +			dom0_event = strtol(optarg, NULL, 10);
> +			break;
>  		}
>  	}
>  	if (optind != argc)
> diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
> index c487089..d3040ba 100644
> --- a/tools/xenstore/xenstored_core.h
> +++ b/tools/xenstore/xenstored_core.h
> @@ -168,6 +168,7 @@ void trace(const char *fmt, ...);
>  void dtrace_io(const struct connection *conn, const struct buffered_data *data, int out);
>  
>  extern int event_fd;
> +extern int dom0_event;
>  
>  /* Map the kernel's xenstore page. */
>  void *xenbus_map(void);
> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
> index aca2149..648eb1d 100644
> --- a/tools/xenstore/xenstored_domain.c
> +++ b/tools/xenstore/xenstored_domain.c
> @@ -593,8 +593,19 @@ void restore_existing_connections(void)
>  }
>  
>  #ifdef __MINIOS__
> -static inline int dom0_init(void) 
> +static int dom0_init(void)
>  {
> +	struct domain *domain;
> +	int domid = 0;
> +	evtchn_port_t port = dom0_event;
> +
> +	domain = new_domain(NULL, domid, port);
> +	domain->interface = xc_gnttab_map_grant_ref(*xcg_handle, domid,
> +			GNTTAB_RESERVED_XENSTORE, PROT_READ|PROT_WRITE);
> +	talloc_steal(domain->conn, domain);
> +
> +	xc_evtchn_notify(xce_handle, domain->port);
> +
>  	return 0;
>  }
>  #else

[PATCH 16/18] xenstored: use domain_is_unprivileged instead of checking conn->id

[Daniel De Graaf]

This centralizes all the permission checking for privileged domains in
preparation for allowing domains other than dom0 to be privileged.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c   |    6 +++---
 tools/xenstore/xenstored_domain.c |    8 ++++----
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index 4ec63f1..eea5fd6 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -488,7 +488,7 @@ static enum xs_perm_type perm_for_conn(struct connection *conn,
 		mask &= ~XS_PERM_WRITE;
 
 	/* Owners and tools get it all... */
-	if (!conn->id || perms[0].id == conn->id
+	if (!domain_is_unprivileged(conn) || perms[0].id == conn->id
                 || (conn->target && perms[0].id == conn->target->id))
 		return (XS_PERM_READ|XS_PERM_WRITE|XS_PERM_OWNER) & mask;
 
@@ -826,11 +826,11 @@ static struct node *construct_node(struct connection *conn, const char *name)
 	node->tdb = tdb_context(conn);
 	node->name = talloc_strdup(node, name);
 
-	/* Inherit permissions, except domains own what they create */
+	/* Inherit permissions, except unprivileged domains own what they create */
 	node->num_perms = parent->num_perms;
 	node->perms = talloc_memdup(node, parent->perms,
 				    node->num_perms * sizeof(node->perms[0]));
-	if (conn && conn->id)
+	if (domain_is_unprivileged(conn))
 		node->perms[0].id = conn->id;
 
 	/* No children, no data */
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 648eb1d..5f4a09e 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -336,7 +336,7 @@ void do_introduce(struct connection *conn, struct buffered_data *in)
 		return;
 	}
 
-	if (conn->id != 0 || !conn->can_write) {
+	if (domain_is_unprivileged(conn) || !conn->can_write) {
 		send_error(conn, EACCES);
 		return;
 	}
@@ -413,7 +413,7 @@ void do_set_target(struct connection *conn, struct buffered_data *in)
 		return;
 	}
 
-	if (conn->id != 0 || !conn->can_write) {
+	if (domain_is_unprivileged(conn) || !conn->can_write) {
 		send_error(conn, EACCES);
 		return;
 	}
@@ -465,7 +465,7 @@ void do_release(struct connection *conn, const char *domid_str)
 		return;
 	}
 
-	if (conn->id != 0) {
+	if (domain_is_unprivileged(conn)) {
 		send_error(conn, EACCES);
 		return;
 	}
@@ -502,7 +502,7 @@ void do_resume(struct connection *conn, const char *domid_str)
 		return;
 	}
 
-	if (conn->id != 0) {
+	if (domain_is_unprivileged(conn)) {
 		send_error(conn, EACCES);
 		return;
 	}
-- 
1.7.7.5

Re: [PATCH 16/18] xenstored: use domain_is_unprivileged instead of checking conn->id

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> This centralizes all the permission checking for privileged domains in
> preparation for allowing domains other than dom0 to be privileged.
> 
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  tools/xenstore/xenstored_core.c   |    6 +++---
>  tools/xenstore/xenstored_domain.c |    8 ++++----
>  2 files changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
> index 4ec63f1..eea5fd6 100644
> --- a/tools/xenstore/xenstored_core.c
> +++ b/tools/xenstore/xenstored_core.c
> @@ -488,7 +488,7 @@ static enum xs_perm_type perm_for_conn(struct connection *conn,
>  		mask &= ~XS_PERM_WRITE;
>  
>  	/* Owners and tools get it all... */
> -	if (!conn->id || perms[0].id == conn->id
> +	if (!domain_is_unprivileged(conn) || perms[0].id == conn->id

domain_is_unprivileged is:
        conn && conn->domain && conn->domain->domid != 0
        
which isn't quite the same as the code being replaced. The difference
appears to be the conn->id is valid for socket connections as well as
domain connections whereas conn->domain is only present for domain
connections.

Does this change not mean that, for the dom0-process xenstored
configuration we now treat socket based connections as unprivileged
where previously they would be unprivileged?


>                  || (conn->target && perms[0].id == conn->target->id))
>  		return (XS_PERM_READ|XS_PERM_WRITE|XS_PERM_OWNER) & mask;
>  
> @@ -826,11 +826,11 @@ static struct node *construct_node(struct connection *conn, const char *name)
>  	node->tdb = tdb_context(conn);
>  	node->name = talloc_strdup(node, name);
>  
> -	/* Inherit permissions, except domains own what they create */
> +	/* Inherit permissions, except unprivileged domains own what they create */
>  	node->num_perms = parent->num_perms;
>  	node->perms = talloc_memdup(node, parent->perms,
>  				    node->num_perms * sizeof(node->perms[0]));
> -	if (conn && conn->id)
> +	if (domain_is_unprivileged(conn))
>  		node->perms[0].id = conn->id;
>  
>  	/* No children, no data */
> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
> index 648eb1d..5f4a09e 100644
> --- a/tools/xenstore/xenstored_domain.c
> +++ b/tools/xenstore/xenstored_domain.c
> @@ -336,7 +336,7 @@ void do_introduce(struct connection *conn, struct buffered_data *in)
>  		return;
>  	}
>  
> -	if (conn->id != 0 || !conn->can_write) {
> +	if (domain_is_unprivileged(conn) || !conn->can_write) {
>  		send_error(conn, EACCES);
>  		return;
>  	}
> @@ -413,7 +413,7 @@ void do_set_target(struct connection *conn, struct buffered_data *in)
>  		return;
>  	}
>  
> -	if (conn->id != 0 || !conn->can_write) {
> +	if (domain_is_unprivileged(conn) || !conn->can_write) {
>  		send_error(conn, EACCES);
>  		return;
>  	}
> @@ -465,7 +465,7 @@ void do_release(struct connection *conn, const char *domid_str)
>  		return;
>  	}
>  
> -	if (conn->id != 0) {
> +	if (domain_is_unprivileged(conn)) {
>  		send_error(conn, EACCES);
>  		return;
>  	}
> @@ -502,7 +502,7 @@ void do_resume(struct connection *conn, const char *domid_str)
>  		return;
>  	}
>  
> -	if (conn->id != 0) {
> +	if (domain_is_unprivileged(conn)) {
>  		send_error(conn, EACCES);
>  		return;
>  	}

Re: [PATCH 16/18] xenstored: use domain_is_unprivileged instead of checking conn->id

[Daniel De Graaf]

On 01/18/2012 06:44 AM, Ian Campbell wrote:
> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
>> This centralizes all the permission checking for privileged domains in
>> preparation for allowing domains other than dom0 to be privileged.
>>
>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>> ---
>>  tools/xenstore/xenstored_core.c   |    6 +++---
>>  tools/xenstore/xenstored_domain.c |    8 ++++----
>>  2 files changed, 7 insertions(+), 7 deletions(-)
>>
>> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
>> index 4ec63f1..eea5fd6 100644
>> --- a/tools/xenstore/xenstored_core.c
>> +++ b/tools/xenstore/xenstored_core.c
>> @@ -488,7 +488,7 @@ static enum xs_perm_type perm_for_conn(struct connection *conn,
>>  		mask &= ~XS_PERM_WRITE;
>>  
>>  	/* Owners and tools get it all... */
>> -	if (!conn->id || perms[0].id == conn->id
>> +	if (!domain_is_unprivileged(conn) || perms[0].id == conn->id
> 
> domain_is_unprivileged is:
>         conn && conn->domain && conn->domain->domid != 0
>         
> which isn't quite the same as the code being replaced. The difference
> appears to be the conn->id is valid for socket connections as well as
> domain connections whereas conn->domain is only present for domain
> connections.
> 
> Does this change not mean that, for the dom0-process xenstored
> configuration we now treat socket based connections as unprivileged
> where previously they would be unprivileged?

No. For dom0 socket connections, conn->domain is NULL so the connection
is not unprivileged (making it privileged). This is why sane people do not
make boolean functions test for "un" cases :)

[PATCH 17/18] xenstored: add --priv-domid parameter

[Daniel De Graaf]

This parameter identifies an alternative service domain which has
superuser access to the xenstore database, which is currently required
to set up a new domain's xenstore entries.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/xenstore/xenstored_core.c   |    5 +++++
 tools/xenstore/xenstored_core.h   |    1 +
 tools/xenstore/xenstored_domain.c |    2 +-
 3 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
index eea5fd6..9d087de 100644
--- a/tools/xenstore/xenstored_core.c
+++ b/tools/xenstore/xenstored_core.c
@@ -1774,6 +1774,7 @@ static struct option options[] = {
 	{ "event", 1, NULL, 'e' },
 	{ "help", 0, NULL, 'H' },
 	{ "no-fork", 0, NULL, 'N' },
+	{ "priv-domid", 1, NULL, 'p' },
 	{ "output-pid", 0, NULL, 'P' },
 	{ "entry-size", 1, NULL, 'S' },
 	{ "trace-file", 1, NULL, 'T' },
@@ -1786,6 +1787,7 @@ static struct option options[] = {
 
 extern void dump_conn(struct connection *conn); 
 int dom0_event = 0;
+int priv_domid = 0;
 
 int main(int argc, char *argv[])
 {
@@ -1852,6 +1854,9 @@ int main(int argc, char *argv[])
 		case 'e':
 			dom0_event = strtol(optarg, NULL, 10);
 			break;
+		case 'p':
+			priv_domid = strtol(optarg, NULL, 10);
+			break;
 		}
 	}
 	if (optind != argc)
diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
index d3040ba..03e2e48 100644
--- a/tools/xenstore/xenstored_core.h
+++ b/tools/xenstore/xenstored_core.h
@@ -169,6 +169,7 @@ void dtrace_io(const struct connection *conn, const struct buffered_data *data,
 
 extern int event_fd;
 extern int dom0_event;
+extern int priv_domid;
 
 /* Map the kernel's xenstore page. */
 void *xenbus_map(void);
diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
index 5f4a09e..46bcf3e 100644
--- a/tools/xenstore/xenstored_domain.c
+++ b/tools/xenstore/xenstored_domain.c
@@ -241,7 +241,7 @@ bool domain_can_read(struct connection *conn)
 
 bool domain_is_unprivileged(struct connection *conn)
 {
-	return (conn && conn->domain && conn->domain->domid != 0);
+	return (conn && conn->domain && conn->domain->domid != 0 && conn->domain->domid != priv_domid);
 }
 
 bool domain_can_write(struct connection *conn)
-- 
1.7.7.5

Re: [PATCH 17/18] xenstored: add --priv-domid parameter

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> This parameter identifies an alternative service domain which has
> superuser access to the xenstore database, which is currently required
> to set up a new domain's xenstore entries.

Is this equivalent to dom0 adding write permissions to various paths for
that domain as it builds it or is there more to it than that.

I know that the determination of "various paths" is non-trivial, so I'm
not actually suggesting that is a better approach.

> 
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  tools/xenstore/xenstored_core.c   |    5 +++++
>  tools/xenstore/xenstored_core.h   |    1 +
>  tools/xenstore/xenstored_domain.c |    2 +-
>  3 files changed, 7 insertions(+), 1 deletions(-)
> 
> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
> index eea5fd6..9d087de 100644
> --- a/tools/xenstore/xenstored_core.c
> +++ b/tools/xenstore/xenstored_core.c
> @@ -1774,6 +1774,7 @@ static struct option options[] = {
>  	{ "event", 1, NULL, 'e' },
>  	{ "help", 0, NULL, 'H' },
>  	{ "no-fork", 0, NULL, 'N' },
> +	{ "priv-domid", 1, NULL, 'p' },
>  	{ "output-pid", 0, NULL, 'P' },
>  	{ "entry-size", 1, NULL, 'S' },
>  	{ "trace-file", 1, NULL, 'T' },
> @@ -1786,6 +1787,7 @@ static struct option options[] = {
>  
>  extern void dump_conn(struct connection *conn); 
>  int dom0_event = 0;
> +int priv_domid = 0;
>  
>  int main(int argc, char *argv[])
>  {
> @@ -1852,6 +1854,9 @@ int main(int argc, char *argv[])
>  		case 'e':
>  			dom0_event = strtol(optarg, NULL, 10);
>  			break;
> +		case 'p':
> +			priv_domid = strtol(optarg, NULL, 10);
> +			break;
>  		}
>  	}
>  	if (optind != argc)
> diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
> index d3040ba..03e2e48 100644
> --- a/tools/xenstore/xenstored_core.h
> +++ b/tools/xenstore/xenstored_core.h
> @@ -169,6 +169,7 @@ void dtrace_io(const struct connection *conn, const struct buffered_data *data,
>  
>  extern int event_fd;
>  extern int dom0_event;
> +extern int priv_domid;
>  
>  /* Map the kernel's xenstore page. */
>  void *xenbus_map(void);
> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
> index 5f4a09e..46bcf3e 100644
> --- a/tools/xenstore/xenstored_domain.c
> +++ b/tools/xenstore/xenstored_domain.c
> @@ -241,7 +241,7 @@ bool domain_can_read(struct connection *conn)
>  
>  bool domain_is_unprivileged(struct connection *conn)
>  {
> -	return (conn && conn->domain && conn->domain->domid != 0);
> +	return (conn && conn->domain && conn->domain->domid != 0 && conn->domain->domid != priv_domid);
>  }
>  
>  bool domain_can_write(struct connection *conn)

Re: [PATCH 17/18] xenstored: add --priv-domid parameter

[Daniel De Graaf]

On 01/18/2012 06:48 AM, Ian Campbell wrote:
> On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
>> This parameter identifies an alternative service domain which has
>> superuser access to the xenstore database, which is currently required
>> to set up a new domain's xenstore entries.
> 
> Is this equivalent to dom0 adding write permissions to various paths for
> that domain as it builds it or is there more to it than that.
> 
> I know that the determination of "various paths" is non-trivial, so I'm
> not actually suggesting that is a better approach.
> 

It's more: the domain builder needs to create entries owned by the new
domain, and similar to UNIX chown() can only be called by the superuser.
The domain builder also currently relies on the fact that new keys it
creates inherit the parent's ownership instead of being owned by dom0.
The introduce operation is also privileged.

>>
>> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
>> ---
>>  tools/xenstore/xenstored_core.c   |    5 +++++
>>  tools/xenstore/xenstored_core.h   |    1 +
>>  tools/xenstore/xenstored_domain.c |    2 +-
>>  3 files changed, 7 insertions(+), 1 deletions(-)
>>
>> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
>> index eea5fd6..9d087de 100644
>> --- a/tools/xenstore/xenstored_core.c
>> +++ b/tools/xenstore/xenstored_core.c
>> @@ -1774,6 +1774,7 @@ static struct option options[] = {
>>  	{ "event", 1, NULL, 'e' },
>>  	{ "help", 0, NULL, 'H' },
>>  	{ "no-fork", 0, NULL, 'N' },
>> +	{ "priv-domid", 1, NULL, 'p' },
>>  	{ "output-pid", 0, NULL, 'P' },
>>  	{ "entry-size", 1, NULL, 'S' },
>>  	{ "trace-file", 1, NULL, 'T' },
>> @@ -1786,6 +1787,7 @@ static struct option options[] = {
>>  
>>  extern void dump_conn(struct connection *conn); 
>>  int dom0_event = 0;
>> +int priv_domid = 0;
>>  
>>  int main(int argc, char *argv[])
>>  {
>> @@ -1852,6 +1854,9 @@ int main(int argc, char *argv[])
>>  		case 'e':
>>  			dom0_event = strtol(optarg, NULL, 10);
>>  			break;
>> +		case 'p':
>> +			priv_domid = strtol(optarg, NULL, 10);
>> +			break;
>>  		}
>>  	}
>>  	if (optind != argc)
>> diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
>> index d3040ba..03e2e48 100644
>> --- a/tools/xenstore/xenstored_core.h
>> +++ b/tools/xenstore/xenstored_core.h
>> @@ -169,6 +169,7 @@ void dtrace_io(const struct connection *conn, const struct buffered_data *data,
>>  
>>  extern int event_fd;
>>  extern int dom0_event;
>> +extern int priv_domid;
>>  
>>  /* Map the kernel's xenstore page. */
>>  void *xenbus_map(void);
>> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
>> index 5f4a09e..46bcf3e 100644
>> --- a/tools/xenstore/xenstored_domain.c
>> +++ b/tools/xenstore/xenstored_domain.c
>> @@ -241,7 +241,7 @@ bool domain_can_read(struct connection *conn)
>>  
>>  bool domain_is_unprivileged(struct connection *conn)
>>  {
>> -	return (conn && conn->domain && conn->domain->domid != 0);
>> +	return (conn && conn->domain && conn->domain->domid != 0 && conn->domain->domid != priv_domid);
>>  }
>>  
>>  bool domain_can_write(struct connection *conn)
> 


-- 
Daniel De Graaf
National Security Agency

Re: [PATCH 17/18] xenstored: add --priv-domid parameter

[Ian Campbell]

On Wed, 2012-01-18 at 14:41 +0000, Daniel De Graaf wrote:
> On 01/18/2012 06:48 AM, Ian Campbell wrote:
> > On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> >> This parameter identifies an alternative service domain which has
> >> superuser access to the xenstore database, which is currently required
> >> to set up a new domain's xenstore entries.
> > 
> > Is this equivalent to dom0 adding write permissions to various paths for
> > that domain as it builds it or is there more to it than that.
> > 
> > I know that the determination of "various paths" is non-trivial, so I'm
> > not actually suggesting that is a better approach.
> > 
> 
> It's more: the domain builder needs to create entries owned by the new
> domain, and similar to UNIX chown() can only be called by the superuser.
> The domain builder also currently relies on the fact that new keys it
> creates inherit the parent's ownership instead of being owned by dom0.
> The introduce operation is also privileged.

Thanks for explaining. I wonder if there is somewhere this can be
usefully written down so that "privileged" is well defined?

docs/misc/xenstore.txt seems to be more about the wire protocol than the
underlying semantics. Perhaps someone on list can suggest a suitable
place?

> 
> >>
> >> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> >> ---
> >>  tools/xenstore/xenstored_core.c   |    5 +++++
> >>  tools/xenstore/xenstored_core.h   |    1 +
> >>  tools/xenstore/xenstored_domain.c |    2 +-
> >>  3 files changed, 7 insertions(+), 1 deletions(-)
> >>
> >> diff --git a/tools/xenstore/xenstored_core.c b/tools/xenstore/xenstored_core.c
> >> index eea5fd6..9d087de 100644
> >> --- a/tools/xenstore/xenstored_core.c
> >> +++ b/tools/xenstore/xenstored_core.c
> >> @@ -1774,6 +1774,7 @@ static struct option options[] = {
> >>  	{ "event", 1, NULL, 'e' },
> >>  	{ "help", 0, NULL, 'H' },
> >>  	{ "no-fork", 0, NULL, 'N' },
> >> +	{ "priv-domid", 1, NULL, 'p' },
> >>  	{ "output-pid", 0, NULL, 'P' },
> >>  	{ "entry-size", 1, NULL, 'S' },
> >>  	{ "trace-file", 1, NULL, 'T' },
> >> @@ -1786,6 +1787,7 @@ static struct option options[] = {
> >>  
> >>  extern void dump_conn(struct connection *conn); 
> >>  int dom0_event = 0;
> >> +int priv_domid = 0;
> >>  
> >>  int main(int argc, char *argv[])
> >>  {
> >> @@ -1852,6 +1854,9 @@ int main(int argc, char *argv[])
> >>  		case 'e':
> >>  			dom0_event = strtol(optarg, NULL, 10);
> >>  			break;
> >> +		case 'p':
> >> +			priv_domid = strtol(optarg, NULL, 10);
> >> +			break;
> >>  		}
> >>  	}
> >>  	if (optind != argc)
> >> diff --git a/tools/xenstore/xenstored_core.h b/tools/xenstore/xenstored_core.h
> >> index d3040ba..03e2e48 100644
> >> --- a/tools/xenstore/xenstored_core.h
> >> +++ b/tools/xenstore/xenstored_core.h
> >> @@ -169,6 +169,7 @@ void dtrace_io(const struct connection *conn, const struct buffered_data *data,
> >>  
> >>  extern int event_fd;
> >>  extern int dom0_event;
> >> +extern int priv_domid;
> >>  
> >>  /* Map the kernel's xenstore page. */
> >>  void *xenbus_map(void);
> >> diff --git a/tools/xenstore/xenstored_domain.c b/tools/xenstore/xenstored_domain.c
> >> index 5f4a09e..46bcf3e 100644
> >> --- a/tools/xenstore/xenstored_domain.c
> >> +++ b/tools/xenstore/xenstored_domain.c
> >> @@ -241,7 +241,7 @@ bool domain_can_read(struct connection *conn)
> >>  
> >>  bool domain_is_unprivileged(struct connection *conn)
> >>  {
> >> -	return (conn && conn->domain && conn->domain->domid != 0);
> >> +	return (conn && conn->domain && conn->domain->domid != 0 && conn->domain->domid != priv_domid);
> >>  }
> >>  
> >>  bool domain_can_write(struct connection *conn)
> > 
> 
> 

[PATCH 18/18] xenstored: Add stub domain builder

[Daniel De Graaf]

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 tools/include/xen-sys/Linux/xenbus_dev.h |   55 +++++++++++++++++
 tools/xenstore/Makefile                  |    9 ++-
 tools/xenstore/init-xenstore-domain.c    |   96 ++++++++++++++++++++++++++++++
 3 files changed, 158 insertions(+), 2 deletions(-)
 create mode 100644 tools/include/xen-sys/Linux/xenbus_dev.h
 create mode 100644 tools/xenstore/init-xenstore-domain.c

diff --git a/tools/include/xen-sys/Linux/xenbus_dev.h b/tools/include/xen-sys/Linux/xenbus_dev.h
new file mode 100644
index 0000000..b107be9
--- /dev/null
+++ b/tools/include/xen-sys/Linux/xenbus_dev.h
@@ -0,0 +1,55 @@
+/******************************************************************************
+ * evtchn.h
+ *
+ * Interface to /dev/xen/xenbus_backend.
+ *
+ * Copyright (c) 2011 Bastian Blank <waldi@debian.org>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version 2
+ * as published by the Free Software Foundation; or, when distributed
+ * separately from the Linux kernel or incorporated into other
+ * software packages, subject to the following license:
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this source file (the "Software"), to deal in the Software without
+ * restriction, including without limitation the rights to use, copy, modify,
+ * merge, publish, distribute, sublicense, and/or sell copies of the Software,
+ * and to permit persons to whom the Software is furnished to do so, subject to
+ * the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef __LINUX_XEN_XENBUS_DEV_H__
+#define __LINUX_XEN_XENBUS_DEV_H__
+
+#include <linux/ioctl.h>
+
+#define IOCTL_XENBUS_BACKEND_EVTCHN			\
+	_IOC(_IOC_NONE, 'B', 0, 0)
+
+#define IOCTL_XENBUS_ALLOC				\
+	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
+struct ioctl_xenbus_alloc {
+	/* IN */
+	/* The domain ID (must exist) for xenstore */
+	uint16_t dom;
+	uint16_t pad;
+	/* OUT */
+	/* The port allocated for xenbus communication */
+	uint32_t port;
+	/* Always set to GNTTAB_RESERVED_XENSTORE */
+	uint32_t grant_ref;
+};
+
+#endif /* __LINUX_XEN_XENBUS_DEV_H__ */
diff --git a/tools/xenstore/Makefile b/tools/xenstore/Makefile
index 3a061d6..9b411a5 100644
--- a/tools/xenstore/Makefile
+++ b/tools/xenstore/Makefile
@@ -26,7 +26,7 @@ LIBXENSTORE := libxenstore.a
 xenstore xenstore-control: CFLAGS += -static
 endif
 
-ALL_TARGETS = libxenstore.so libxenstore.a clients xs_tdb_dump xenstored
+ALL_TARGETS = libxenstore.so libxenstore.a clients xs_tdb_dump xenstored init-xenstore-domain
 
 ifdef CONFIG_STUBDOM
 CFLAGS += -DNO_SOCKETS=1 -DNO_LOCAL_XENBUS=1 -DNO_SYSLOG=1 -DNO_REOPEN_LOG=1
@@ -50,6 +50,11 @@ xenstored_probes.o: xenstored_solaris.o
 CFLAGS += -DHAVE_DTRACE=1
 endif
 
+init-xenstore-domain.o: CFLAGS += $(CFLAGS_libxenguest)
+
+init-xenstore-domain: init-xenstore-domain.o $(LIBXENSTORE)
+	$(CC) $(LDFLAGS) $^ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) -o $@ $(APPEND_LDFLAGS)
+
 xenstored: $(XENSTORED_OBJS)
 	$(CC) $(LDFLAGS) $^ $(LDLIBS_libxenctrl) $(SOCKET_LIBS) -o $@ $(APPEND_LDFLAGS)
 
@@ -85,7 +90,7 @@ libxenstore.a: xs.o xs_lib.o
 clean:
 	rm -f *.a *.o *.opic *.so* xenstored_probes.h
 	rm -f xenstored xs_random xs_stress xs_crashme
-	rm -f xs_tdb_dump xenstore-control
+	rm -f xs_tdb_dump xenstore-control init-xenstore-domain
 	rm -f xenstore $(CLIENTS)
 	$(RM) $(DEPS)
 
diff --git a/tools/xenstore/init-xenstore-domain.c b/tools/xenstore/init-xenstore-domain.c
new file mode 100644
index 0000000..30f6c9b
--- /dev/null
+++ b/tools/xenstore/init-xenstore-domain.c
@@ -0,0 +1,96 @@
+#include <fcntl.h>
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <xenctrl.h>
+#include <xc_dom.h>
+#include <xs.h>
+#include <xen/sys/xenbus_dev.h>
+
+static uint32_t domid = -1;
+
+static int build(xc_interface *xch, char** argv)
+{
+	char cmdline[512];
+	uint32_t ssid;
+	xen_domain_handle_t handle = { 0 };
+	int rv;
+	int xs_fd = open("/dev/xen/xenbus_backend", O_RDWR);
+	struct xc_dom_image *dom;
+	int maxmem = atoi(argv[2]);
+	int limit_kb = (maxmem + 1)*1024;
+	struct ioctl_xenbus_alloc alloc;
+
+	rv = xc_flask_context_to_sid(xch, argv[3], strlen(argv[3]), &ssid);
+	if (rv) return rv;
+	rv = xc_domain_create(xch, ssid, handle, 0, &domid);
+	if (rv) return rv;
+	rv = xc_domain_max_vcpus(xch, domid, 1);
+	if (rv) return rv;
+	rv = xc_domain_setmaxmem(xch, domid, limit_kb);
+	if (rv) return rv;
+	rv = xc_domain_set_memmap_limit(xch, domid, limit_kb);
+	if (rv) return rv;
+
+	alloc.dom = domid;
+	rv = ioctl(xs_fd, IOCTL_XENBUS_ALLOC, &alloc);
+	if (rv) return rv;
+	snprintf(cmdline, 512, "--event %d", alloc.port);
+
+	dom = xc_dom_allocate(xch, cmdline, NULL);
+	rv = xc_dom_kernel_file(dom, argv[1]);
+	if (rv) return rv;
+	rv = xc_dom_boot_xen_init(dom, xch, domid);
+	if (rv) return rv;
+	rv = xc_dom_parse_image(dom);
+	if (rv) return rv;
+	rv = xc_dom_mem_init(dom, maxmem);
+	if (rv) return rv;
+	rv = xc_dom_boot_mem_init(dom);
+	if (rv) return rv;
+	rv = xc_dom_build_image(dom);
+	if (rv) return rv;
+	rv = xc_dom_boot_image(dom);
+	if (rv) return rv;
+
+	xc_dom_release(dom);
+
+	rv = xc_domain_set_virq_handler(xch, domid, VIRQ_DOM_EXC);
+	if (rv) return rv;
+	rv = xc_domain_unpause(xch, domid);
+	if (rv) return rv;
+
+	return 0;
+}
+
+int main(int argc, char** argv)
+{
+	xc_interface *xch;
+	struct xs_handle *xsh;
+	char buf[16];
+	int rv;
+
+	if (argc != 4) {
+		printf("Use: %s <xenstore-kernel> <memory_mb> <flask-label>\n", argv[0]);
+		return 2;
+	}
+
+	xch = xc_interface_open(NULL, NULL, 0);
+	if (!xch) return 1;
+
+	rv = build(xch, argv);
+
+	xc_interface_close(xch);
+
+	if (rv) return 1;
+
+	xsh = xs_open(0);
+	rv = snprintf(buf, 16, "%d", domid);
+	xs_write(xsh, XBT_NULL, "/tool/xenstored/domid", buf, rv);
+	xs_daemon_close(xsh);
+
+	return 0;
+}
-- 
1.7.7.5

Re: [PATCH 18/18] xenstored: Add stub domain builder

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>

I think eventually we should aim to have this be a rough drop in
replacement for /usr/sbin/*xenstored (similar params where it makes
sense etc) but this will do for now:

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/include/xen-sys/Linux/xenbus_dev.h |   55 +++++++++++++++++
>  tools/xenstore/Makefile                  |    9 ++-
>  tools/xenstore/init-xenstore-domain.c    |   96 ++++++++++++++++++++++++++++++
>  3 files changed, 158 insertions(+), 2 deletions(-)
>  create mode 100644 tools/include/xen-sys/Linux/xenbus_dev.h
>  create mode 100644 tools/xenstore/init-xenstore-domain.c
> 
> diff --git a/tools/include/xen-sys/Linux/xenbus_dev.h b/tools/include/xen-sys/Linux/xenbus_dev.h
> new file mode 100644
> index 0000000..b107be9
> --- /dev/null
> +++ b/tools/include/xen-sys/Linux/xenbus_dev.h
> @@ -0,0 +1,55 @@
> +/******************************************************************************
> + * evtchn.h
> + *
> + * Interface to /dev/xen/xenbus_backend.
> + *
> + * Copyright (c) 2011 Bastian Blank <waldi@debian.org>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License version 2
> + * as published by the Free Software Foundation; or, when distributed
> + * separately from the Linux kernel or incorporated into other
> + * software packages, subject to the following license:
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this source file (the "Software"), to deal in the Software without
> + * restriction, including without limitation the rights to use, copy, modify,
> + * merge, publish, distribute, sublicense, and/or sell copies of the Software,
> + * and to permit persons to whom the Software is furnished to do so, subject to
> + * the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
> + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
> + * IN THE SOFTWARE.
> + */
> +
> +#ifndef __LINUX_XEN_XENBUS_DEV_H__
> +#define __LINUX_XEN_XENBUS_DEV_H__
> +
> +#include <linux/ioctl.h>
> +
> +#define IOCTL_XENBUS_BACKEND_EVTCHN			\
> +	_IOC(_IOC_NONE, 'B', 0, 0)
> +
> +#define IOCTL_XENBUS_ALLOC				\
> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
> +struct ioctl_xenbus_alloc {
> +	/* IN */
> +	/* The domain ID (must exist) for xenstore */
> +	uint16_t dom;
> +	uint16_t pad;
> +	/* OUT */
> +	/* The port allocated for xenbus communication */
> +	uint32_t port;
> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
> +	uint32_t grant_ref;
> +};
> +
> +#endif /* __LINUX_XEN_XENBUS_DEV_H__ */
> diff --git a/tools/xenstore/Makefile b/tools/xenstore/Makefile
> index 3a061d6..9b411a5 100644
> --- a/tools/xenstore/Makefile
> +++ b/tools/xenstore/Makefile
> @@ -26,7 +26,7 @@ LIBXENSTORE := libxenstore.a
>  xenstore xenstore-control: CFLAGS += -static
>  endif
>  
> -ALL_TARGETS = libxenstore.so libxenstore.a clients xs_tdb_dump xenstored
> +ALL_TARGETS = libxenstore.so libxenstore.a clients xs_tdb_dump xenstored init-xenstore-domain
>  
>  ifdef CONFIG_STUBDOM
>  CFLAGS += -DNO_SOCKETS=1 -DNO_LOCAL_XENBUS=1 -DNO_SYSLOG=1 -DNO_REOPEN_LOG=1
> @@ -50,6 +50,11 @@ xenstored_probes.o: xenstored_solaris.o
>  CFLAGS += -DHAVE_DTRACE=1
>  endif
>  
> +init-xenstore-domain.o: CFLAGS += $(CFLAGS_libxenguest)
> +
> +init-xenstore-domain: init-xenstore-domain.o $(LIBXENSTORE)
> +	$(CC) $(LDFLAGS) $^ $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) -o $@ $(APPEND_LDFLAGS)
> +
>  xenstored: $(XENSTORED_OBJS)
>  	$(CC) $(LDFLAGS) $^ $(LDLIBS_libxenctrl) $(SOCKET_LIBS) -o $@ $(APPEND_LDFLAGS)
>  
> @@ -85,7 +90,7 @@ libxenstore.a: xs.o xs_lib.o
>  clean:
>  	rm -f *.a *.o *.opic *.so* xenstored_probes.h
>  	rm -f xenstored xs_random xs_stress xs_crashme
> -	rm -f xs_tdb_dump xenstore-control
> +	rm -f xs_tdb_dump xenstore-control init-xenstore-domain
>  	rm -f xenstore $(CLIENTS)
>  	$(RM) $(DEPS)
>  
> diff --git a/tools/xenstore/init-xenstore-domain.c b/tools/xenstore/init-xenstore-domain.c
> new file mode 100644
> index 0000000..30f6c9b
> --- /dev/null
> +++ b/tools/xenstore/init-xenstore-domain.c
> @@ -0,0 +1,96 @@
> +#include <fcntl.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <stdint.h>
> +#include <stdlib.h>
> +#include <sys/ioctl.h>
> +#include <sys/mman.h>
> +#include <xenctrl.h>
> +#include <xc_dom.h>
> +#include <xs.h>
> +#include <xen/sys/xenbus_dev.h>
> +
> +static uint32_t domid = -1;
> +
> +static int build(xc_interface *xch, char** argv)
> +{
> +	char cmdline[512];
> +	uint32_t ssid;
> +	xen_domain_handle_t handle = { 0 };
> +	int rv;
> +	int xs_fd = open("/dev/xen/xenbus_backend", O_RDWR);
> +	struct xc_dom_image *dom;
> +	int maxmem = atoi(argv[2]);
> +	int limit_kb = (maxmem + 1)*1024;
> +	struct ioctl_xenbus_alloc alloc;
> +
> +	rv = xc_flask_context_to_sid(xch, argv[3], strlen(argv[3]), &ssid);
> +	if (rv) return rv;
> +	rv = xc_domain_create(xch, ssid, handle, 0, &domid);
> +	if (rv) return rv;
> +	rv = xc_domain_max_vcpus(xch, domid, 1);
> +	if (rv) return rv;
> +	rv = xc_domain_setmaxmem(xch, domid, limit_kb);
> +	if (rv) return rv;
> +	rv = xc_domain_set_memmap_limit(xch, domid, limit_kb);
> +	if (rv) return rv;
> +
> +	alloc.dom = domid;
> +	rv = ioctl(xs_fd, IOCTL_XENBUS_ALLOC, &alloc);
> +	if (rv) return rv;
> +	snprintf(cmdline, 512, "--event %d", alloc.port);
> +
> +	dom = xc_dom_allocate(xch, cmdline, NULL);
> +	rv = xc_dom_kernel_file(dom, argv[1]);
> +	if (rv) return rv;
> +	rv = xc_dom_boot_xen_init(dom, xch, domid);
> +	if (rv) return rv;
> +	rv = xc_dom_parse_image(dom);
> +	if (rv) return rv;
> +	rv = xc_dom_mem_init(dom, maxmem);
> +	if (rv) return rv;
> +	rv = xc_dom_boot_mem_init(dom);
> +	if (rv) return rv;
> +	rv = xc_dom_build_image(dom);
> +	if (rv) return rv;
> +	rv = xc_dom_boot_image(dom);
> +	if (rv) return rv;
> +
> +	xc_dom_release(dom);
> +
> +	rv = xc_domain_set_virq_handler(xch, domid, VIRQ_DOM_EXC);
> +	if (rv) return rv;
> +	rv = xc_domain_unpause(xch, domid);
> +	if (rv) return rv;
> +
> +	return 0;
> +}
> +
> +int main(int argc, char** argv)
> +{
> +	xc_interface *xch;
> +	struct xs_handle *xsh;
> +	char buf[16];
> +	int rv;
> +
> +	if (argc != 4) {
> +		printf("Use: %s <xenstore-kernel> <memory_mb> <flask-label>\n", argv[0]);
> +		return 2;
> +	}
> +
> +	xch = xc_interface_open(NULL, NULL, 0);
> +	if (!xch) return 1;
> +
> +	rv = build(xch, argv);
> +
> +	xc_interface_close(xch);
> +
> +	if (rv) return 1;
> +
> +	xsh = xs_open(0);
> +	rv = snprintf(buf, 16, "%d", domid);
> +	xs_write(xsh, XBT_NULL, "/tool/xenstored/domid", buf, rv);
> +	xs_daemon_close(xsh);
> +
> +	return 0;
> +}

[PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

This adds an ioctl to the /dev/xen/xenbus_backend device allowing the
xenbus backend to be started after the kernel has booted. This is
intended to allow dom0 to start another domain to run xenstore.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 drivers/xen/xenbus/xenbus_comms.c       |    6 ++++
 drivers/xen/xenbus/xenbus_comms.h       |    1 +
 drivers/xen/xenbus/xenbus_dev_backend.c |   44 +++++++++++++++++++++++++++++++
 include/xen/grant_table.h               |    2 +
 include/xen/xenbus_dev.h                |   14 ++++++++++
 5 files changed, 67 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
index 2eff7a6..52fe7ad 100644
--- a/drivers/xen/xenbus/xenbus_comms.c
+++ b/drivers/xen/xenbus/xenbus_comms.c
@@ -234,3 +234,9 @@ int xb_init_comms(void)
 
 	return 0;
 }
+
+void xb_deinit_comms(void)
+{
+	unbind_from_irqhandler(xenbus_irq, &xb_waitq);
+	xenbus_irq = 0;
+}
diff --git a/drivers/xen/xenbus/xenbus_comms.h b/drivers/xen/xenbus/xenbus_comms.h
index 6e42800..c8abd3b 100644
--- a/drivers/xen/xenbus/xenbus_comms.h
+++ b/drivers/xen/xenbus/xenbus_comms.h
@@ -35,6 +35,7 @@
 
 int xs_init(void);
 int xb_init_comms(void);
+void xb_deinit_comms(void);
 
 /* Low level routines. */
 int xb_write(const void *data, unsigned len);
diff --git a/drivers/xen/xenbus/xenbus_dev_backend.c b/drivers/xen/xenbus/xenbus_dev_backend.c
index 3d3be78..854315c 100644
--- a/drivers/xen/xenbus/xenbus_dev_backend.c
+++ b/drivers/xen/xenbus/xenbus_dev_backend.c
@@ -8,7 +8,11 @@
 
 #include <xen/xen.h>
 #include <xen/page.h>
+#include <xen/xenbus.h>
 #include <xen/xenbus_dev.h>
+#include <xen/grant_table.h>
+#include <xen/events.h>
+#include <asm/xen/hypervisor.h>
 
 #include "xenbus_comms.h"
 
@@ -22,6 +26,43 @@ static int xenbus_backend_open(struct inode *inode, struct file *filp)
 	return nonseekable_open(inode, filp);
 }
 
+static long xenbus_alloc(void __user * data)
+{
+	struct ioctl_xenbus_alloc op;
+	struct evtchn_alloc_unbound arg;
+	int err;
+
+	if (copy_from_user(&op, data, sizeof(op)))
+		return -EFAULT;
+
+	gnttab_grant_foreign_access_ref(GNTTAB_RESERVED_XENSTORE, op.dom,
+			virt_to_mfn(xen_store_interface), 0 /* writable */);
+
+	arg.dom = DOMID_SELF;
+	arg.remote_dom = op.dom;
+
+	err = HYPERVISOR_event_channel_op(EVTCHNOP_alloc_unbound, &arg);
+	if (err)
+		return err;
+
+	op.port = arg.port;
+	op.grant_ref = GNTTAB_RESERVED_XENSTORE;
+
+	xs_suspend();
+
+	if (xen_store_evtchn > 0)
+		xb_deinit_comms();
+
+	xen_store_evtchn = arg.port;
+
+	xs_resume();
+
+	if (copy_to_user(data, &op, sizeof(op)))
+		return -EFAULT;
+
+	return 0;
+}
+
 static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned long data)
 {
 	if (!capable(CAP_SYS_ADMIN))
@@ -33,6 +74,9 @@ static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned l
 				return xen_store_evtchn;
 			return -ENODEV;
 
+		case IOCTL_XENBUS_ALLOC:
+			return xenbus_alloc((void __user *)data);
+
 		default:
 			return -ENOTTY;
 	}
diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
index 15f8a00..11e27c3 100644
--- a/include/xen/grant_table.h
+++ b/include/xen/grant_table.h
@@ -46,6 +46,8 @@
 
 #include <xen/features.h>
 
+#define GNTTAB_RESERVED_XENSTORE 1
+
 /* NR_GRANT_FRAMES must be less than or equal to that configured in Xen */
 #define NR_GRANT_FRAMES 4
 
diff --git a/include/xen/xenbus_dev.h b/include/xen/xenbus_dev.h
index ac5f0fe..b107be9 100644
--- a/include/xen/xenbus_dev.h
+++ b/include/xen/xenbus_dev.h
@@ -38,4 +38,18 @@
 #define IOCTL_XENBUS_BACKEND_EVTCHN			\
 	_IOC(_IOC_NONE, 'B', 0, 0)
 
+#define IOCTL_XENBUS_ALLOC				\
+	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
+struct ioctl_xenbus_alloc {
+	/* IN */
+	/* The domain ID (must exist) for xenstore */
+	uint16_t dom;
+	uint16_t pad;
+	/* OUT */
+	/* The port allocated for xenbus communication */
+	uint32_t port;
+	/* Always set to GNTTAB_RESERVED_XENSTORE */
+	uint32_t grant_ref;
+};
+
 #endif /* __LINUX_XEN_XENBUS_DEV_H__ */
-- 
1.7.7.5

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Jan Beulich]

>>> On 13.01.12 at 00:36, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> --- a/include/xen/xenbus_dev.h
> +++ b/include/xen/xenbus_dev.h
> @@ -38,4 +38,18 @@
>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>  	_IOC(_IOC_NONE, 'B', 0, 0)
>  
> +#define IOCTL_XENBUS_ALLOC				\
> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
> +struct ioctl_xenbus_alloc {
> +	/* IN */
> +	/* The domain ID (must exist) for xenstore */
> +	uint16_t dom;
> +	uint16_t pad;
> +	/* OUT */
> +	/* The port allocated for xenbus communication */
> +	uint32_t port;
> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
> +	uint32_t grant_ref;
> +};

As said in my reply to the previous patch version - if the functionality
differs, the number *and* name should be different from the legacy
implementation's. Otherwise, how should compatible user space code
be written?

Jan

> +
>  #endif /* __LINUX_XEN_XENBUS_DEV_H__ */

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

On 01/13/2012 03:20 AM, Jan Beulich wrote:
>>>> On 13.01.12 at 00:36, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>> --- a/include/xen/xenbus_dev.h
>> +++ b/include/xen/xenbus_dev.h
>> @@ -38,4 +38,18 @@
>>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>>  	_IOC(_IOC_NONE, 'B', 0, 0)
>>  
>> +#define IOCTL_XENBUS_ALLOC				\
>> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
>> +struct ioctl_xenbus_alloc {
>> +	/* IN */
>> +	/* The domain ID (must exist) for xenstore */
>> +	uint16_t dom;
>> +	uint16_t pad;
>> +	/* OUT */
>> +	/* The port allocated for xenbus communication */
>> +	uint32_t port;
>> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
>> +	uint32_t grant_ref;
>> +};
> 
> As said in my reply to the previous patch version - if the functionality
> differs, the number *and* name should be different from the legacy
> implementation's. Otherwise, how should compatible user space code
> be written?
> 
> Jan
> 

This implementation has the same functionality as the legacy ioctl,
although it has a different number and is performed on a different file,
so it is already impossible to make automatically compatible userspace
code. The structure name was changed to match other ioctl arguments, but
the contents should be the same as in the legacy ioctl. If changing what
file the ioctl is performed on justifies changing the ioctl name, then I
would prefer the simpler interface where the domain ID is passed in as the
ioctl parameter instead of a structure that only has one useful output.

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Jan Beulich]

>>> On 13.01.12 at 15:06, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> On 01/13/2012 03:20 AM, Jan Beulich wrote:
>>>>> On 13.01.12 at 00:36, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>> --- a/include/xen/xenbus_dev.h
>>> +++ b/include/xen/xenbus_dev.h
>>> @@ -38,4 +38,18 @@
>>>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>>>  	_IOC(_IOC_NONE, 'B', 0, 0)
>>>  
>>> +#define IOCTL_XENBUS_ALLOC				\
>>> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
>>> +struct ioctl_xenbus_alloc {
>>> +	/* IN */
>>> +	/* The domain ID (must exist) for xenstore */
>>> +	uint16_t dom;
>>> +	uint16_t pad;
>>> +	/* OUT */
>>> +	/* The port allocated for xenbus communication */
>>> +	uint32_t port;
>>> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
>>> +	uint32_t grant_ref;
>>> +};
>> 
>> As said in my reply to the previous patch version - if the functionality
>> differs, the number *and* name should be different from the legacy
>> implementation's. Otherwise, how should compatible user space code
>> be written?
>> 
>> Jan
>> 
> 
> This implementation has the same functionality as the legacy ioctl,

It doesn't - none of the "OUT" fields above exist there.

> although it has a different number and is performed on a different file,
> so it is already impossible to make automatically compatible userspace
> code.

That's not what I was aiming at. The problem you're introducing is
that you name the ioctl IOCTL_XENBUS_ALLOC, identical to what
the legacy code named it. Hence one won't be able to write user
mode code to invoke, depending on runtime determination, either
the old or the new function.

Jan

> The structure name was changed to match other ioctl arguments, but
> the contents should be the same as in the legacy ioctl. If changing what
> file the ioctl is performed on justifies changing the ioctl name, then I
> would prefer the simpler interface where the domain ID is passed in as the
> ioctl parameter instead of a structure that only has one useful output.

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

On 01/13/2012 10:37 AM, Jan Beulich wrote:
>>>> On 13.01.12 at 15:06, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>> On 01/13/2012 03:20 AM, Jan Beulich wrote:
>>>>>> On 13.01.12 at 00:36, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>>> --- a/include/xen/xenbus_dev.h
>>>> +++ b/include/xen/xenbus_dev.h
>>>> @@ -38,4 +38,18 @@
>>>>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>>>>  	_IOC(_IOC_NONE, 'B', 0, 0)
>>>>  
>>>> +#define IOCTL_XENBUS_ALLOC				\
>>>> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
>>>> +struct ioctl_xenbus_alloc {
>>>> +	/* IN */
>>>> +	/* The domain ID (must exist) for xenstore */
>>>> +	uint16_t dom;
>>>> +	uint16_t pad;
>>>> +	/* OUT */
>>>> +	/* The port allocated for xenbus communication */
>>>> +	uint32_t port;
>>>> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
>>>> +	uint32_t grant_ref;
>>>> +};
>>>
>>> As said in my reply to the previous patch version - if the functionality
>>> differs, the number *and* name should be different from the legacy
>>> implementation's. Otherwise, how should compatible user space code
>>> be written?
>>>
>>> Jan
>>>
>>
>> This implementation has the same functionality as the legacy ioctl,
> 
> It doesn't - none of the "OUT" fields above exist there.

Are you looking at the same legacy ioctl as I am? I found it in 2.6.18.hg
where the structure was defined as:

typedef struct xenbus_alloc {
    domid_t dom;
    __u32 port;
    __u32 grant_ref;
} xenbus_alloc_t;

The port and grant_ref fields were outputs. I added padding for clarity
since domid_t is uint16_t.

> 
>> although it has a different number and is performed on a different file,
>> so it is already impossible to make automatically compatible userspace
>> code.
> 
> That's not what I was aiming at. The problem you're introducing is
> that you name the ioctl IOCTL_XENBUS_ALLOC, identical to what
> the legacy code named it. Hence one won't be able to write user
> mode code to invoke, depending on runtime determination, either
> the old or the new function.
> 
> Jan
> 
>> The structure name was changed to match other ioctl arguments, but
>> the contents should be the same as in the legacy ioctl. If changing what
>> file the ioctl is performed on justifies changing the ioctl name, then I
>> would prefer the simpler interface where the domain ID is passed in as the
>> ioctl parameter instead of a structure that only has one useful output.
> 
> 

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Jan Beulich]

>>> On 13.01.12 at 16:44, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> On 01/13/2012 10:37 AM, Jan Beulich wrote:
>>>>> On 13.01.12 at 15:06, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>> On 01/13/2012 03:20 AM, Jan Beulich wrote:
>>>>>>> On 13.01.12 at 00:36, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>>>> --- a/include/xen/xenbus_dev.h
>>>>> +++ b/include/xen/xenbus_dev.h
>>>>> @@ -38,4 +38,18 @@
>>>>>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>>>>>  	_IOC(_IOC_NONE, 'B', 0, 0)
>>>>>  
>>>>> +#define IOCTL_XENBUS_ALLOC				\
>>>>> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
>>>>> +struct ioctl_xenbus_alloc {
>>>>> +	/* IN */
>>>>> +	/* The domain ID (must exist) for xenstore */
>>>>> +	uint16_t dom;
>>>>> +	uint16_t pad;
>>>>> +	/* OUT */
>>>>> +	/* The port allocated for xenbus communication */
>>>>> +	uint32_t port;
>>>>> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
>>>>> +	uint32_t grant_ref;
>>>>> +};
>>>>
>>>> As said in my reply to the previous patch version - if the functionality
>>>> differs, the number *and* name should be different from the legacy
>>>> implementation's. Otherwise, how should compatible user space code
>>>> be written?
>>>>
>>>> Jan
>>>>
>>>
>>> This implementation has the same functionality as the legacy ioctl,
>> 
>> It doesn't - none of the "OUT" fields above exist there.
> 
> Are you looking at the same legacy ioctl as I am? I found it in 2.6.18.hg
> where the structure was defined as:
> 
> typedef struct xenbus_alloc {
>     domid_t dom;
>     __u32 port;
>     __u32 grant_ref;
> } xenbus_alloc_t;
> 
> The port and grant_ref fields were outputs. I added padding for clarity
> since domid_t is uint16_t.

Ooops, I'm sorry, indeed - I didn't look at the names and types
closely enough. They were too different, so I assumed the whole
thing is different.

That doesn't eliminate the difference between the two
IOCTL_XENBUS_ALLOC definitions, though
[_IOC(_IOC_NONE, 'X', 0, sizeof(xenbus_alloc_t)) vs
_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))] - one
still can't use both in the same source file.

Additionally (forgive if I'm not up to date in this respect) - is it okay
these days to use uintX_t in exported Linux headers, and then
without including the header that would define the necessary
types?

Jan

>>> although it has a different number and is performed on a different file,
>>> so it is already impossible to make automatically compatible userspace
>>> code.
>> 
>> That's not what I was aiming at. The problem you're introducing is
>> that you name the ioctl IOCTL_XENBUS_ALLOC, identical to what
>> the legacy code named it. Hence one won't be able to write user
>> mode code to invoke, depending on runtime determination, either
>> the old or the new function.
>> 
>> Jan
>> 
>>> The structure name was changed to match other ioctl arguments, but
>>> the contents should be the same as in the legacy ioctl. If changing what
>>> file the ioctl is performed on justifies changing the ioctl name, then I
>>> would prefer the simpler interface where the domain ID is passed in as the
>>> ioctl parameter instead of a structure that only has one useful output.
>> 
>> 

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

On 01/13/2012 11:00 AM, Jan Beulich wrote:
>>>> On 13.01.12 at 16:44, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>> On 01/13/2012 10:37 AM, Jan Beulich wrote:
>>>>>> On 13.01.12 at 15:06, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>>> On 01/13/2012 03:20 AM, Jan Beulich wrote:
>>>>>>>> On 13.01.12 at 00:36, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>>>>> --- a/include/xen/xenbus_dev.h
>>>>>> +++ b/include/xen/xenbus_dev.h
>>>>>> @@ -38,4 +38,18 @@
>>>>>>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>>>>>>  	_IOC(_IOC_NONE, 'B', 0, 0)
>>>>>>  
>>>>>> +#define IOCTL_XENBUS_ALLOC				\
>>>>>> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
>>>>>> +struct ioctl_xenbus_alloc {
>>>>>> +	/* IN */
>>>>>> +	/* The domain ID (must exist) for xenstore */
>>>>>> +	uint16_t dom;
>>>>>> +	uint16_t pad;
>>>>>> +	/* OUT */
>>>>>> +	/* The port allocated for xenbus communication */
>>>>>> +	uint32_t port;
>>>>>> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
>>>>>> +	uint32_t grant_ref;
>>>>>> +};
>>>>>
>>>>> As said in my reply to the previous patch version - if the functionality
>>>>> differs, the number *and* name should be different from the legacy
>>>>> implementation's. Otherwise, how should compatible user space code
>>>>> be written?
>>>>>
>>>>> Jan
>>>>>
>>>>
>>>> This implementation has the same functionality as the legacy ioctl,
>>>
>>> It doesn't - none of the "OUT" fields above exist there.
>>
>> Are you looking at the same legacy ioctl as I am? I found it in 2.6.18.hg
>> where the structure was defined as:
>>
>> typedef struct xenbus_alloc {
>>     domid_t dom;
>>     __u32 port;
>>     __u32 grant_ref;
>> } xenbus_alloc_t;
>>
>> The port and grant_ref fields were outputs. I added padding for clarity
>> since domid_t is uint16_t.
> 
> Ooops, I'm sorry, indeed - I didn't look at the names and types
> closely enough. They were too different, so I assumed the whole
> thing is different.
> 
> That doesn't eliminate the difference between the two
> IOCTL_XENBUS_ALLOC definitions, though
> [_IOC(_IOC_NONE, 'X', 0, sizeof(xenbus_alloc_t)) vs
> _IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))] - one
> still can't use both in the same source file.

Is compatibility with 2.6.18-xen more important than keeping the ioctl
numbers in a single file consistent? The definition in 2.6.18 is also
incorrect (as was my definition) - since the argument is used, it needs
to use _IOWR() or _IOC_READ|_IOC_WRITE instead of _IOC_NONE.

Since this ioctl must already be performed a different file than the
legacy ioctl, a new name seems to be the best solution.

> Additionally (forgive if I'm not up to date in this respect) - is it okay
> these days to use uintX_t in exported Linux headers, and then
> without including the header that would define the necessary
> types?
> 
> Jan

It's done in a number of other header files (I was copying from gntdev.h)
but that doesn't mean it's officially okay. Using __u32 is probably safer.

> 
>>>> although it has a different number and is performed on a different file,
>>>> so it is already impossible to make automatically compatible userspace
>>>> code.
>>>
>>> That's not what I was aiming at. The problem you're introducing is
>>> that you name the ioctl IOCTL_XENBUS_ALLOC, identical to what
>>> the legacy code named it. Hence one won't be able to write user
>>> mode code to invoke, depending on runtime determination, either
>>> the old or the new function.
>>>
>>> Jan
>>>
>>>> The structure name was changed to match other ioctl arguments, but
>>>> the contents should be the same as in the legacy ioctl. If changing what
>>>> file the ioctl is performed on justifies changing the ioctl name, then I
>>>> would prefer the simpler interface where the domain ID is passed in as the
>>>> ioctl parameter instead of a structure that only has one useful output.
>>>

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Jan Beulich]

>>> On 13.01.12 at 18:42, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
> On 01/13/2012 11:00 AM, Jan Beulich wrote:
>>>>> On 13.01.12 at 16:44, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>> On 01/13/2012 10:37 AM, Jan Beulich wrote:
>>>>>>> On 13.01.12 at 15:06, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>>>> On 01/13/2012 03:20 AM, Jan Beulich wrote:
>>>>>>>>> On 13.01.12 at 00:36, Daniel De Graaf <dgdegra@tycho.nsa.gov> wrote:
>>>>>>> --- a/include/xen/xenbus_dev.h
>>>>>>> +++ b/include/xen/xenbus_dev.h
>>>>>>> @@ -38,4 +38,18 @@
>>>>>>>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>>>>>>>  	_IOC(_IOC_NONE, 'B', 0, 0)
>>>>>>>  
>>>>>>> +#define IOCTL_XENBUS_ALLOC				\
>>>>>>> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
>>>>>>> +struct ioctl_xenbus_alloc {
>>>>>>> +	/* IN */
>>>>>>> +	/* The domain ID (must exist) for xenstore */
>>>>>>> +	uint16_t dom;
>>>>>>> +	uint16_t pad;
>>>>>>> +	/* OUT */
>>>>>>> +	/* The port allocated for xenbus communication */
>>>>>>> +	uint32_t port;
>>>>>>> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
>>>>>>> +	uint32_t grant_ref;
>>>>>>> +};
>>>>>>
>>>>>> As said in my reply to the previous patch version - if the functionality
>>>>>> differs, the number *and* name should be different from the legacy
>>>>>> implementation's. Otherwise, how should compatible user space code
>>>>>> be written?
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>
>>>>> This implementation has the same functionality as the legacy ioctl,
>>>>
>>>> It doesn't - none of the "OUT" fields above exist there.
>>>
>>> Are you looking at the same legacy ioctl as I am? I found it in 2.6.18.hg
>>> where the structure was defined as:
>>>
>>> typedef struct xenbus_alloc {
>>>     domid_t dom;
>>>     __u32 port;
>>>     __u32 grant_ref;
>>> } xenbus_alloc_t;
>>>
>>> The port and grant_ref fields were outputs. I added padding for clarity
>>> since domid_t is uint16_t.
>> 
>> Ooops, I'm sorry, indeed - I didn't look at the names and types
>> closely enough. They were too different, so I assumed the whole
>> thing is different.
>> 
>> That doesn't eliminate the difference between the two
>> IOCTL_XENBUS_ALLOC definitions, though
>> [_IOC(_IOC_NONE, 'X', 0, sizeof(xenbus_alloc_t)) vs
>> _IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))] - one
>> still can't use both in the same source file.
> 
> Is compatibility with 2.6.18-xen more important than keeping the ioctl
> numbers in a single file consistent? The definition in 2.6.18 is also
> incorrect (as was my definition) - since the argument is used, it needs
> to use _IOWR() or _IOC_READ|_IOC_WRITE instead of _IOC_NONE.

Hmm, that would mean that apart from IOCTL_EVTCHN_RESET *all*
current ioctl number definitions under include/xen/ are wrong. Very
bad, and ugly to deal with. With generic ioctl handling code not
depending on this, I wonder though whether it's worth fixing at all.
Are you aware of any user mode code depending on the correctness
of these bits?

Which of course isn't to say that it shouldn't be done right for new
ones...

> Since this ioctl must already be performed a different file than the
> legacy ioctl, a new name seems to be the best solution.

That's what I was trying to ask you to do.

Jan

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Ian Campbell]

On Thu, 2012-01-12 at 23:36 +0000, Daniel De Graaf wrote:
> This adds an ioctl to the /dev/xen/xenbus_backend device allowing the
> xenbus backend to be started after the kernel has booted. This is
> intended to allow dom0 to start another domain to run xenstore.

Does this new ioctl need to be made oneshot? What happens if you call it
twice?

Also it should be disabled once a local xenstored has connected
shouldn't it?

Diego's original patch handled that sort of thing, any reason not to
simply forward port it?

> 
> Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
> ---
>  drivers/xen/xenbus/xenbus_comms.c       |    6 ++++
>  drivers/xen/xenbus/xenbus_comms.h       |    1 +
>  drivers/xen/xenbus/xenbus_dev_backend.c |   44 +++++++++++++++++++++++++++++++
>  include/xen/grant_table.h               |    2 +
>  include/xen/xenbus_dev.h                |   14 ++++++++++
>  5 files changed, 67 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
> index 2eff7a6..52fe7ad 100644
> --- a/drivers/xen/xenbus/xenbus_comms.c
> +++ b/drivers/xen/xenbus/xenbus_comms.c
> @@ -234,3 +234,9 @@ int xb_init_comms(void)
>  
>  	return 0;
>  }
> +
> +void xb_deinit_comms(void)
> +{
> +	unbind_from_irqhandler(xenbus_irq, &xb_waitq);
> +	xenbus_irq = 0;
> +}
> diff --git a/drivers/xen/xenbus/xenbus_comms.h b/drivers/xen/xenbus/xenbus_comms.h
> index 6e42800..c8abd3b 100644
> --- a/drivers/xen/xenbus/xenbus_comms.h
> +++ b/drivers/xen/xenbus/xenbus_comms.h
> @@ -35,6 +35,7 @@
>  
>  int xs_init(void);
>  int xb_init_comms(void);
> +void xb_deinit_comms(void);
>  
>  /* Low level routines. */
>  int xb_write(const void *data, unsigned len);
> diff --git a/drivers/xen/xenbus/xenbus_dev_backend.c b/drivers/xen/xenbus/xenbus_dev_backend.c
> index 3d3be78..854315c 100644
> --- a/drivers/xen/xenbus/xenbus_dev_backend.c
> +++ b/drivers/xen/xenbus/xenbus_dev_backend.c
> @@ -8,7 +8,11 @@
>  
>  #include <xen/xen.h>
>  #include <xen/page.h>
> +#include <xen/xenbus.h>
>  #include <xen/xenbus_dev.h>
> +#include <xen/grant_table.h>
> +#include <xen/events.h>
> +#include <asm/xen/hypervisor.h>
>  
>  #include "xenbus_comms.h"
>  
> @@ -22,6 +26,43 @@ static int xenbus_backend_open(struct inode *inode, struct file *filp)
>  	return nonseekable_open(inode, filp);
>  }
>  
> +static long xenbus_alloc(void __user * data)
> +{
> +	struct ioctl_xenbus_alloc op;
> +	struct evtchn_alloc_unbound arg;
> +	int err;
> +
> +	if (copy_from_user(&op, data, sizeof(op)))
> +		return -EFAULT;
> +
> +	gnttab_grant_foreign_access_ref(GNTTAB_RESERVED_XENSTORE, op.dom,
> +			virt_to_mfn(xen_store_interface), 0 /* writable */);
> +
> +	arg.dom = DOMID_SELF;
> +	arg.remote_dom = op.dom;
> +
> +	err = HYPERVISOR_event_channel_op(EVTCHNOP_alloc_unbound, &arg);
> +	if (err)
> +		return err;
> +
> +	op.port = arg.port;
> +	op.grant_ref = GNTTAB_RESERVED_XENSTORE;
> +
> +	xs_suspend();
> +
> +	if (xen_store_evtchn > 0)
> +		xb_deinit_comms();
> +
> +	xen_store_evtchn = arg.port;
> +
> +	xs_resume();
> +
> +	if (copy_to_user(data, &op, sizeof(op)))
> +		return -EFAULT;
> +
> +	return 0;
> +}
> +
>  static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned long data)
>  {
>  	if (!capable(CAP_SYS_ADMIN))
> @@ -33,6 +74,9 @@ static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned l
>  				return xen_store_evtchn;
>  			return -ENODEV;
>  
> +		case IOCTL_XENBUS_ALLOC:
> +			return xenbus_alloc((void __user *)data);
> +
>  		default:
>  			return -ENOTTY;
>  	}
> diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
> index 15f8a00..11e27c3 100644
> --- a/include/xen/grant_table.h
> +++ b/include/xen/grant_table.h
> @@ -46,6 +46,8 @@
>  
>  #include <xen/features.h>
>  
> +#define GNTTAB_RESERVED_XENSTORE 1
> +
>  /* NR_GRANT_FRAMES must be less than or equal to that configured in Xen */
>  #define NR_GRANT_FRAMES 4
>  
> diff --git a/include/xen/xenbus_dev.h b/include/xen/xenbus_dev.h
> index ac5f0fe..b107be9 100644
> --- a/include/xen/xenbus_dev.h
> +++ b/include/xen/xenbus_dev.h
> @@ -38,4 +38,18 @@
>  #define IOCTL_XENBUS_BACKEND_EVTCHN			\
>  	_IOC(_IOC_NONE, 'B', 0, 0)
>  
> +#define IOCTL_XENBUS_ALLOC				\
> +	_IOC(_IOC_NONE, 'B', 1, sizeof(struct ioctl_xenbus_alloc))
> +struct ioctl_xenbus_alloc {
> +	/* IN */
> +	/* The domain ID (must exist) for xenstore */
> +	uint16_t dom;
> +	uint16_t pad;
> +	/* OUT */
> +	/* The port allocated for xenbus communication */
> +	uint32_t port;
> +	/* Always set to GNTTAB_RESERVED_XENSTORE */
> +	uint32_t grant_ref;
> +};
> +
>  #endif /* __LINUX_XEN_XENBUS_DEV_H__ */

Re: [PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

On 01/18/2012 07:07 AM, Ian Campbell wrote:
> On Thu, 2012-01-12 at 23:36 +0000, Daniel De Graaf wrote:
>> This adds an ioctl to the /dev/xen/xenbus_backend device allowing the
>> xenbus backend to be started after the kernel has booted. This is
>> intended to allow dom0 to start another domain to run xenstore.
> 
> Does this new ioctl need to be made oneshot? What happens if you call it
> twice?
> 
> Also it should be disabled once a local xenstored has connected
> shouldn't it?
> 
> Diego's original patch handled that sort of thing, any reason not to
> simply forward port it?
> 

My original patch was trying to be more flexible in allowing xenstored
to be created later (migrating watches), or restarted in a different
domain. However, since the current version lacks that ability the
oneshot safeguards from Diego's patch should probably be added.

Re: [PATCH v2 00/18] Xenstore stub domain

[Ian Campbell]

On Thu, 2012-01-12 at 23:35 +0000, Daniel De Graaf wrote:
> Changes from v1:
>  - set_virq_handler implemented in libxc
>  - added custom domain builder for xenstored
>  - xenstore/console domain IDs now pulled from xenstore
>  - migration support when using split xenstored (untested, should work)
>  - slightly less intrusive NO_SOCKETS xenstored patch
>    (still has many ifdefs to avoid pulling in socket headers or symbols)
>  - virq handlers removed from dying domain when clearing event channels
>  - dummy XSM module restricts getdomaininfo similar to no-XSM case
>  - formatting/type fixups
>  - partial ioctl compatibility with legacy IOCTL_XENBUS_ALLOC
>    (the new device path uses 'B' not 'X' as the base so the ioctl number
>    does not match; otherwise, should be compatible).
> 
> To start xenstored, run:
> 
> tools/xenstore/init-xenstore-domain stubdom/mini-os-x86_64-xenstore/mini-os 20 system_u:system_r:domU_t
> 
> This will populate the xenstore domid key /tool/xenstore/domid

I'm just about to go through and review this properly but from the above
it's sounding good, thanks!

You have rather undermined my intentions for the next hackathon
though ;-) I was planning to investigate and reinvigorate this stuff, I
guess there is still stub-oxenstored left for me though!

Speaking of which, will you be at the hackathon?
http://lists.xen.org/archives/html/xen-devel/2012-01/msg00268.html

Ian.


> 
> ----
> 
> [PATCH 01/18] xen: reinstate previously unused hypercall
> [PATCH 02/18] xen: allow global VIRQ handlers to be delegated to
> [PATCH 03/18] xen: use XSM instead of IS_PRIV for getdomaininfo
> [PATCH 04/18] xen: Preserve reserved grant entries when switching
> 
> [PATCH 05/18] tools/libxl: pull xenstore/console domids from
> [PATCH 06/18] lib{xc,xl}: Seed grant tables with xenstore and
> 
> [PATCH 07/18] mini-os: avoid crash if no console is provided
> [PATCH 08/18] mini-os: avoid crash if no xenstore is provided
> [PATCH 09/18] mini-os: remove per-fd evtchn limit
> [PATCH 10/18] xenstored: use grant references instead of
> [PATCH 11/18] xenstored: add NO_SOCKETS compilation option
> [PATCH 12/18] xenstored support for in-memory rather than FS based
> [PATCH 13/18] xenstored: support running in minios stubdom
> [PATCH 14/18] xenstored: always use xc_gnttab_munmap in stubdom
> [PATCH 15/18] xenstored: add --event parameter for bootstrapping
> Removed old #16; shared page is no longer used to pass event.
> [PATCH 16/18] xenstored: use domain_is_unprivileged instead of
> [PATCH 17/18] xenstored: add --priv-domid parameter
> 
> [PATCH 18/18] xenstored: Add stub domain builder
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

[PATCH v3 00/21] Xenstore stub domain

[Daniel De Graaf]

Changes from v2:
 - configuration support added to mini-os build system
 - add mini-os support for conditionally compiling frontends, xenbus
 - XENMEM_remove_from_physmap moved out of arch-specific code
 - use uint32_t for virqs
 - warn when dropping grant v2-only flags when switching versions
 - IOCTL_XENBUS_BACKEND_SETUP name changed so userspace can implement compat
 - ioctl now returns -EEXIST if xenstored has already been connected
 - various cosmetic cleanups, shuffling

Changes from v1:
 - set_virq_handler implemented in libxc
 - added custom domain builder for xenstored
 - xenstore/console domain IDs now pulled from xenstore
 - migration support when using split xenstored (untested, should work)
 - slightly less intrusive NO_SOCKETS xenstored patch
   (still has many ifdefs to avoid pulling in socket headers or symbols)
 - virq handlers removed from dying domain when clearing event channels
 - dummy XSM module restricts getdomaininfo similar to no-XSM case
 - formatting/type fixups
 - partial ioctl compatibility with legacy IOCTL_XENBUS_ALLOC

To start xenstored, run:

tools/xenstore/init-xenstore-domain stubdom/mini-os-x86_64-xenstore/mini-os 20 system_u:system_r:domU_t

This will populate the xenstore domid key /tool/xenstore/domid

Other notes:

The console for xenstored is not set up by init-xenstore-domain. If the
hypervisor is compiled with VERBOSE or debug=y, it will be visible on
the hypervisor serial console (or ring buffer if enabled with
console_to_ring). The xenstore stub domain itself supports console
output, and init-xenstore-domain could be extended to daemonize and
spool this output to a log file. The normal xenconsole daemon cannot be
used here due to the possibility of a deadlock.

----

[PATCH 01/21] xen: reinstate previously unused
[PATCH 02/21] xen: allow global VIRQ handlers to be delegated to
[PATCH 03/21] xen: change virq parameters from int to uint32_t
 - new in v3: cleanup as suggested by Jan Beulich
[PATCH 04/21] xen: use XSM instead of IS_PRIV for getdomaininfo
[PATCH 05/21] xen: Preserve reserved grant entries when switching

[PATCH 06/21] tools/libxl: pull xenstore/console domids from
[PATCH 07/21] lib{xc,xl}: Seed grant tables with xenstore and

[PATCH 08/21] mini-os: avoid crash if no console is provided
[PATCH 09/21] mini-os: remove per-fd evtchn limit
[PATCH 10/21] mini-os: create app-specific configuration
[PATCH 11/21] mini-os: make frontends and xenbus optional
[PATCH 12/21] mini-os: fix list.h include guard name
 - #10-12 are new in v3, replace v2's #8 and part of #13

[PATCH 13/21] xenstored: use grant references instead of
[PATCH 14/21] xenstored: add NO_SOCKETS compilation option
[PATCH 15/21] xenstored support for in-memory rather than FS based
[PATCH 16/21] xenstored: support running in minios stubdom
[PATCH 17/21] stubdom: enable xenstored build
[PATCH 18/21] xenstored: add --event parameter for bootstrapping
[PATCH 19/21] xenstored: use domain_is_unprivileged instead of
[PATCH 20/21] xenstored: add --priv-domid parameter
[PATCH 21/21] xenstored: Add stub domain builder

[PATCH] xenbus: Add support for xenbus backend in stub domain

[PATCH] xenbus: Add support for xenbus backend in stub domain

[Daniel De Graaf]

This adds an ioctl to the /dev/xen/xenbus_backend device allowing the
xenbus backend to be started after the kernel has booted. This is
intended to allow dom0 to start another domain to run xenstore.

Signed-off-by: Daniel De Graaf <dgdegra@tycho.nsa.gov>
---
 drivers/xen/xenbus/xenbus_comms.c       |    6 ++++
 drivers/xen/xenbus/xenbus_comms.h       |    1 +
 drivers/xen/xenbus/xenbus_dev_backend.c |   51 +++++++++++++++++++++++++++++++
 include/xen/grant_table.h               |    2 +
 include/xen/xenbus_dev.h                |    3 ++
 5 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/drivers/xen/xenbus/xenbus_comms.c b/drivers/xen/xenbus/xenbus_comms.c
index 2eff7a6..52fe7ad 100644
--- a/drivers/xen/xenbus/xenbus_comms.c
+++ b/drivers/xen/xenbus/xenbus_comms.c
@@ -234,3 +234,9 @@ int xb_init_comms(void)
 
 	return 0;
 }
+
+void xb_deinit_comms(void)
+{
+	unbind_from_irqhandler(xenbus_irq, &xb_waitq);
+	xenbus_irq = 0;
+}
diff --git a/drivers/xen/xenbus/xenbus_comms.h b/drivers/xen/xenbus/xenbus_comms.h
index 6e42800..c8abd3b 100644
--- a/drivers/xen/xenbus/xenbus_comms.h
+++ b/drivers/xen/xenbus/xenbus_comms.h
@@ -35,6 +35,7 @@
 
 int xs_init(void);
 int xb_init_comms(void);
+void xb_deinit_comms(void);
 
 /* Low level routines. */
 int xb_write(const void *data, unsigned len);
diff --git a/drivers/xen/xenbus/xenbus_dev_backend.c b/drivers/xen/xenbus/xenbus_dev_backend.c
index 3d3be78..be738c4 100644
--- a/drivers/xen/xenbus/xenbus_dev_backend.c
+++ b/drivers/xen/xenbus/xenbus_dev_backend.c
@@ -8,7 +8,11 @@
 
 #include <xen/xen.h>
 #include <xen/page.h>
+#include <xen/xenbus.h>
 #include <xen/xenbus_dev.h>
+#include <xen/grant_table.h>
+#include <xen/events.h>
+#include <asm/xen/hypervisor.h>
 
 #include "xenbus_comms.h"
 
@@ -22,6 +26,50 @@ static int xenbus_backend_open(struct inode *inode, struct file *filp)
 	return nonseekable_open(inode, filp);
 }
 
+static long xenbus_alloc(domid_t domid)
+{
+	struct evtchn_alloc_unbound arg;
+	int err = -EEXIST;
+
+	xs_suspend();
+
+	/* If xenstored_ready is nonzero, that means we have already talked to
+	 * xenstore and set up watches. These watches will be restored by
+	 * xs_resume, but that requires communication over the port established
+	 * below that is not visible to anyone until the ioctl returns.
+	 *
+	 * This can be resolved by splitting the ioctl into two parts
+	 * (postponing the resume until xenstored is active) but this is
+	 * unnecessarily complex for the intended use where xenstored is only
+	 * started once - so return -EEXIST if it's already running.
+	 */
+	if (xenstored_ready)
+		goto out_err;
+
+	gnttab_grant_foreign_access_ref(GNTTAB_RESERVED_XENSTORE, domid,
+			virt_to_mfn(xen_store_interface), 0 /* writable */);
+
+	arg.dom = DOMID_SELF;
+	arg.remote_dom = domid;
+
+	err = HYPERVISOR_event_channel_op(EVTCHNOP_alloc_unbound, &arg);
+	if (err)
+		goto out_err;
+
+	if (xen_store_evtchn > 0)
+		xb_deinit_comms();
+
+	xen_store_evtchn = arg.port;
+
+	xs_resume();
+
+	return arg.port;
+
+ out_err:
+	xs_suspend_cancel();
+	return err;
+}
+
 static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned long data)
 {
 	if (!capable(CAP_SYS_ADMIN))
@@ -33,6 +81,9 @@ static long xenbus_backend_ioctl(struct file *file, unsigned int cmd, unsigned l
 				return xen_store_evtchn;
 			return -ENODEV;
 
+		case IOCTL_XENBUS_BACKEND_SETUP:
+			return xenbus_alloc(data);
+
 		default:
 			return -ENOTTY;
 	}
diff --git a/include/xen/grant_table.h b/include/xen/grant_table.h
index 15f8a00..11e27c3 100644
--- a/include/xen/grant_table.h
+++ b/include/xen/grant_table.h
@@ -46,6 +46,8 @@
 
 #include <xen/features.h>
 
+#define GNTTAB_RESERVED_XENSTORE 1
+
 /* NR_GRANT_FRAMES must be less than or equal to that configured in Xen */
 #define NR_GRANT_FRAMES 4
 
diff --git a/include/xen/xenbus_dev.h b/include/xen/xenbus_dev.h
index ac5f0fe..bbee8c6 100644
--- a/include/xen/xenbus_dev.h
+++ b/include/xen/xenbus_dev.h
@@ -38,4 +38,7 @@
 #define IOCTL_XENBUS_BACKEND_EVTCHN			\
 	_IOC(_IOC_NONE, 'B', 0, 0)
 
+#define IOCTL_XENBUS_BACKEND_SETUP			\
+	_IOC(_IOC_NONE, 'B', 1, 0)
+
 #endif /* __LINUX_XEN_XENBUS_DEV_H__ */
-- 
1.7.7.5