[PATCH v4 00/31] libxl child process handling

[PATCH v4 00/31] libxl child process handling

[Ian Jackson]

This series has now been tested and is in a suitable form for review
and, if thought fit, committing to xen-unstable.  It still culminates
in a patch to convert bootloader execution to the new event machinery.

Changes since v3 ("[RFC PATCH 00/20]") include:
 - All comments on previous version addressed.
 - Several new bugfix patches etc., marked with * below.
 - Broke the bootloader execution rewrite up into three patches.
 - Testing and consequent fixes (only to previously un-acked patches);
   in particular, the final patch has some significant changes.

Bugfixes for problems reported by Roger Pau Monne:
  02/31 libxl: ao: allow immediate completion
  03/31 libxl: fix hang due to libxl__initiate_device_remove
  04/31 libxl: Fix eventloop_iteration over-locking
* 05/31 libxl: remove poller from list in libxl__poller_get

Other general bugfixes:
* 01/31 .gitignore: Add a missing file
  06/31 libxl: Fix leak of ctx->lock
  07/31 tools: Correct PTHREAD options in config/StdGNU.mk
  08/31 libxl: Use PTHREAD_CFLAGS, LDFLAGS, LIBS
  09/31 tools: Use PTHREAD_CFLAGS, _LDFLAGS, _LIBS
  26/31 libxl: Clean up setdefault in do_domain_create
  30/31 libxl: make libxl_create_logfile const-correct

Clarifications and improvements related to memory allocation:
  10/31 libxl: Crash (more sensibly) on malloc failure
  11/31 libxl: Make libxl__zalloc et al tolerate a NULL gc

Preparatory work:
  12/31 libxl: Introduce some convenience macros
  13/31 libxl: include <ctype.h> and introduce CTYPE helper macro
  14/31 libxl: Provide libxl_string_list_length
  15/31 libxl: include <_libxl_paths.h> in libxl_internal.h
  16/31 libxl: abolish libxl_ctx_postfork
* 23/31 autoconf: New test for openpty et al.
* 24/31 libxl: provide libxl__remove_file et al.
* 25/31 libxl: Introduce libxl__sendmsg_fds and libxl__recvmsg_fds

Event-related infrastructure and fixes:
  17/31 libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS
  18/31 libxl: Protect fds with CLOEXEC even with forking threads
* 19/31 libxl: provide STATE_AO_GC
* 20/31 libxl: handle POLLERR, POLLHUP, POLLNVAL properly
* 21/31 libxl: support multiple libxl__ev_fds for the same fd
  22/31 libxl: event API: new facilities for waiting for subprocesses
  27/31 libxl: provide libxl__datacopier_*
  28/31 libxl: provide libxl__openpty_*
  29/31 libxl: ao: Convert libxl_run_bootloader
* 31/31 libxl: log bootloader output

[PATCH 01/31] .gitignore: Add a missing file

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 .gitignore |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/.gitignore b/.gitignore
index 2782ee5..cd12b56 100644
--- a/.gitignore
+++ b/.gitignore
@@ -357,6 +357,7 @@ tools/firmware/etherboot/eb-roms.h
 tools/firmware/etherboot/gpxe-git-snapshot.tar.gz
 tools/misc/xenwatchdogd
 tools/misc/xen-hvmcrash
+tools/misc/xen-lowmemd
 tools/libvchan/vchan-node[12]
 tools/ocaml/*/.ocamldep.make
 tools/ocaml/*/*.cm[ixao]
-- 
1.7.2.5

[PATCH 02/31] libxl: ao: allow immediate completion

[Ian Jackson]

Make it possible to complete an ao during its initating function.

Previously this was not generally possible because initiators did not
have an egc.  But there is no reason why an ao initiator should not
have an egc, so make the standard macros provide one.

Change the internal documentation comments accordingly.  (This change,
which means that an initiator function may call a completion callback
directly, is already consistent with the documented external API.)

We also invent of a new state flag "constructing" which indicates
whether we are between ao__create and ao__inprogress.  This is a
slightly optimisation which allows ao_complete to not bother poking
the wakeup pipe, since the logic in ao__inprogress will not run the
event loop if the ao is complete on entry.

Also fix the wording in the libxl_internal.h comment forbidding use of
ao_how-taking functions from within libxl.  (There are sadly currently
some such functions.)

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl_event.c    |    7 ++++++-
 tools/libxl/libxl_internal.h |   14 ++++++++++----
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 271949a..c89add8 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -1225,7 +1225,9 @@ void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc)
 
     if (ao->poller) {
         assert(ao->in_initiator);
-        libxl__poller_wakeup(egc, ao->poller);
+        if (!ao->constructing)
+            /* don't bother with this if we're not in the event loop */
+            libxl__poller_wakeup(egc, ao->poller);
     } else if (ao->how.callback) {
         LIBXL_TAILQ_INSERT_TAIL(&egc->aos_for_callback, ao, entry_for_callback);
     } else {
@@ -1251,6 +1253,7 @@ libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid,
     if (!ao) goto out;
 
     ao->magic = LIBXL__AO_MAGIC;
+    ao->constructing = 1;
     ao->in_initiator = 1;
     ao->poller = 0;
     ao->domid = domid;
@@ -1275,7 +1278,9 @@ int libxl__ao_inprogress(libxl__ao *ao)
     int rc;
 
     assert(ao->magic == LIBXL__AO_MAGIC);
+    assert(ao->constructing);
     assert(ao->in_initiator);
+    ao->constructing = 0;
 
     if (ao->poller) {
         /* Caller wants it done synchronously. */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e0a1070..b1e0588 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -347,7 +347,7 @@ struct libxl__egc {
 
 struct libxl__ao {
     uint32_t magic;
-    unsigned in_initiator:1, complete:1, notified:1;
+    unsigned constructing:1, in_initiator:1, complete:1, notified:1;
     int rc;
     libxl__gc gc;
     libxl_asyncop_how how;
@@ -1209,7 +1209,11 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
  * operation ("ao") machinery.  The function should take a parameter
  * const libxl_asyncop_how *ao_how and must start with a call to
  * AO_INITIATOR_ENTRY.  These functions MAY NOT be called from
- * outside libxl, because they can cause reentrancy callbacks.
+ * inside libxl, because they can cause reentrancy callbacks.
+ *
+ * For the same reason functions taking an ao_how may make themselves
+ * an egc with EGC_INIT (and they will generally want to, to be able
+ * to immediately complete an ao during its setup).
  *
  * Lifecycle of an ao:
  *
@@ -1240,8 +1244,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
  *   directly or indirectly, should call libxl__ao_complete (with the
  *   ctx locked, as it will generally already be in any event callback
  *   function).  This must happen exactly once for each ao (and not if
- *   the ao has been destroyed, obviously), and it may not happen
- *   until libxl__ao_inprogress has been called on the ao.
+ *   the ao has been destroyed, obviously).
  *
  * - Note that during callback functions, two gcs are available:
  *    - The one in egc, whose lifetime is only this callback
@@ -1255,12 +1258,14 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
     libxl__ctx_lock(ctx);                                       \
     libxl__ao *ao = libxl__ao_create(ctx, domid, ao_how);       \
     if (!ao) { libxl__ctx_unlock(ctx); return ERROR_NOMEM; }    \
+    libxl__egc egc[1]; LIBXL_INIT_EGC(egc[0],ctx);              \
     AO_GC;
 
 #define AO_INPROGRESS ({                                        \
         libxl_ctx *ao__ctx = libxl__gc_owner(&ao->gc);          \
         int ao__rc = libxl__ao_inprogress(ao);                  \
         libxl__ctx_unlock(ao__ctx); /* gc is now invalid */     \
+        EGC_FREE;                                               \
         (ao__rc);                                               \
    })
 
@@ -1269,6 +1274,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
         assert(rc);                                             \
         libxl__ao_abort(ao);                                    \
         libxl__ctx_unlock(ao__ctx); /* gc is now invalid */     \
+        EGC_FREE;                                               \
         (rc);                                                   \
     })
 
-- 
1.7.2.5

[PATCH 03/31] libxl: fix hang due to libxl__initiate_device_remove

[Ian Jackson]

libxl__initiate_device_remove might discover that the operation was
complete, immediately (typically, if the device is already removed).

Previously, in this situation, it would return 0 to the caller but
never call libxl__ao_complete.  Fix this.  This necessitates passing
the egc in from the functions which are the ao initiators.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl.c          |    8 ++++----
 tools/libxl/libxl_device.c   |   18 ++++++++++++------
 tools/libxl/libxl_internal.h |    3 ++-
 3 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 7a54524..dd948a8 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -1406,7 +1406,7 @@ int libxl_device_disk_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_disk(gc, domid, disk, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
@@ -1873,7 +1873,7 @@ int libxl_device_nic_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_nic(gc, domid, nic, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
@@ -2220,7 +2220,7 @@ int libxl_device_vkb_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_vkb(gc, domid, vkb, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
@@ -2353,7 +2353,7 @@ int libxl_device_vfb_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_vfb(gc, domid, vfb, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index c2880e0..c7e057d 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -376,7 +376,8 @@ static void device_remove_callback(libxl__egc *egc, libxl__ev_devstate *ds,
     device_remove_cleanup(gc, aorm);
 }
 
-int libxl__initiate_device_remove(libxl__ao *ao, libxl__device *dev)
+int libxl__initiate_device_remove(libxl__egc *egc, libxl__ao *ao,
+                                  libxl__device *dev)
 {
     AO_GC;
     libxl_ctx *ctx = libxl__gc_owner(gc);
@@ -388,11 +389,11 @@ int libxl__initiate_device_remove(libxl__ao *ao, libxl__device *dev)
     libxl__ao_device_remove *aorm = 0;
 
     if (!state)
-        goto out;
+        goto out_ok;
     if (atoi(state) != 4) {
         libxl__device_destroy_tapdisk(gc, be_path);
         xs_rm(ctx->xsh, XBT_NULL, be_path);
-        goto out;
+        goto out_ok;
     }
 
 retry_transaction:
@@ -404,7 +405,7 @@ retry_transaction:
             goto retry_transaction;
         else {
             rc = ERROR_FAIL;
-            goto out;
+            goto out_fail;
         }
     }
 
@@ -417,13 +418,18 @@ retry_transaction:
     rc = libxl__ev_devstate_wait(gc, &aorm->ds, device_remove_callback,
                                  state_path, XenbusStateClosed,
                                  LIBXL_DESTROY_TIMEOUT * 1000);
-    if (rc) goto out;
+    if (rc) goto out_fail;
 
     return 0;
 
- out:
+ out_fail:
+    assert(rc);
     device_remove_cleanup(gc, aorm);
     return rc;
+
+ out_ok:
+    libxl__ao_complete(egc, ao, 0);
+    return 0;
 }
 
 int libxl__device_destroy(libxl__gc *gc, libxl__device *dev)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index b1e0588..5167b71 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -692,7 +692,8 @@ _hidden int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state);
  * this is done, the ao will be completed.  An error
  * return from libxl__initiate_device_remove means that the ao
  * will _not_ be completed and the caller must do so. */
-_hidden int libxl__initiate_device_remove(libxl__ao*, libxl__device *dev);
+_hidden int libxl__initiate_device_remove(libxl__egc*, libxl__ao*,
+                                          libxl__device *dev);
 
 /*
  * libxl__ev_devstate - waits a given time for a device to
-- 
1.7.2.5

[PATCH 04/31] libxl: Fix eventloop_iteration over-locking

[Ian Jackson]

eventloop_iteration's head comment says that it must be called with
the ctx locked exactly once, and this is indeed true, and it's done
correctly at both the call sites.

However, it takes out the lock an additional time itself.  This is
wrong because it prevents the unlocks around poll from being
effective.  This would mean that a multithreaded event-loop using
program might suffer from undesired blocking, as one thread trying to
enter libxl might end up stalled by another thread waiting for a slow
event.  So remove those two lock calls.

Also add a couple of comments documenting the locking behaviour of
libxl__ao_inprogress and libxl__egc_cleanup.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl_event.c    |    4 ----
 tools/libxl/libxl_internal.h |    4 ++--
 2 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index c89add8..5ac6334 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -1058,8 +1058,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
     int rc;
     struct timeval now;
     
-    CTX_LOCK;
-
     rc = libxl__gettimeofday(gc, &now);
     if (rc) goto out;
 
@@ -1102,8 +1100,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
     afterpoll_internal(egc, poller,
                        poller->fd_polls_allocd, poller->fd_polls, now);
 
-    CTX_UNLOCK;
-
     rc = 0;
  out:
     return rc;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5167b71..04f13f6 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1191,7 +1191,7 @@ libxl__device_model_version_running(libxl__gc *gc, uint32_t domid);
 _hidden void libxl__egc_cleanup(libxl__egc *egc);
   /* Frees memory allocated within this egc's gc, and and report all
    * occurred events via callback, if applicable.  May reenter the
-   * application; see restrictions above. */
+   * application; see restrictions above.  The ctx must be UNLOCKED. */
 
 /* convenience macros: */
 
@@ -1287,7 +1287,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
  * libxl__ao_inprogress MUST be called with the ctx locked exactly once. */
 _hidden libxl__ao *libxl__ao_create(libxl_ctx*, uint32_t domid,
                                     const libxl_asyncop_how*);
-_hidden int libxl__ao_inprogress(libxl__ao *ao);
+_hidden int libxl__ao_inprogress(libxl__ao *ao); /* temporarily unlocks */
 _hidden void libxl__ao_abort(libxl__ao *ao);
 _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
 
-- 
1.7.2.5

[PATCH 05/31] libxl: remove poller from list in libxl__poller_get

[Ian Jackson]

From: Roger Pau Monne <roger.pau@citrix.com>

Remove poller from the list once it has been requested.
Fixes a double-free bug.

Signed-off-by: Roger Pau Monne <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@citrix.com>
---
 tools/libxl/libxl_event.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 5ac6334..9cb172a 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -1010,8 +1010,10 @@ libxl__poller *libxl__poller_get(libxl_ctx *ctx)
     int rc;
 
     libxl__poller *p = LIBXL_LIST_FIRST(&ctx->pollers_idle);
-    if (p)
+    if (p) {
+        LIBXL_LIST_REMOVE(p, entry);
         return p;
+    }
 
     p = malloc(sizeof(*p));
     if (!p) {
-- 
1.7.2.5

[PATCH 06/31] libxl: Fix leak of ctx->lock

[Ian Jackson]

A mutex created with pthread_mutex_init, like ctx->lock, may need to
be destroyed with pthread_mutex_destroy.

Also, previously, if libxl__init_recursive_mutex failed, the nascent
ctx would be leaked.  Add some comments which will hopefully make
these kind of mistakes less likely in future.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl.c |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index dd948a8..f41b62f 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -39,10 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     memset(ctx, 0, sizeof(libxl_ctx));
     ctx->lg = lg;
 
-    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
-        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
-        return ERROR_FAIL;
-    }
+    /* First initialise pointers (cannot fail) */
 
     LIBXL_TAILQ_INIT(&ctx->occurred);
 
@@ -61,6 +58,16 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     LIBXL_TAILQ_INIT(&ctx->death_list);
     libxl__ev_xswatch_init(&ctx->death_watch);
 
+    /* The mutex is special because we can't idempotently destroy it */
+
+    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
+        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
+        free(ctx);
+        ctx = 0;
+    }
+
+    /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
+
     rc = libxl__poller_init(ctx, &ctx->poller_app);
     if (rc) goto out;
 
@@ -150,6 +157,8 @@ int libxl_ctx_free(libxl_ctx *ctx)
 
     discard_events(&ctx->occurred);
 
+    pthread_mutex_destroy(&ctx->lock);
+
     GC_FREE;
     free(ctx);
     return 0;
-- 
1.7.2.5

[PATCH 07/31] tools: Correct PTHREAD options in config/StdGNU.mk

[Ian Jackson]

It is not correct to say -lpthread.  The correct option is -pthread,
which may have sundry other effects on code generation etc.  It needs
to be passed both to compilation and linking.

Fix the configure test to test -pthread, and plumb the resulting flag
through to PTHREAD_{CFLAGS,LDFLAGS} in Tools.mk; also substitute
PTHREAD_LIBS (although this will currently always be empty).
Remove PTHREAD_LIBS setting from StdGNU.mk.

Fix the one user (libxc) to use PTHREAD_{CFLAGS,LDFLAGS} too.

There are still some other users in tree which pass -pthread or
-lpthread by adding it as a literal to their own compiler options.
These will be fixed in a later patch.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
Acked-by: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 config/StdGNU.mk     |    1 -
 config/Tools.mk.in   |    4 ++
 tools/configure      |  101 ++++++++++++++++++++++++++++++++++++--------------
 tools/configure.ac   |    5 +-
 tools/libxc/Makefile |    4 +-
 tools/m4/pthread.m4  |   41 ++++++++++++++++++++
 tools/m4/savevar.m4  |    6 +++
 7 files changed, 130 insertions(+), 32 deletions(-)
 create mode 100644 tools/m4/pthread.m4
 create mode 100644 tools/m4/savevar.m4

diff --git a/config/StdGNU.mk b/config/StdGNU.mk
index e2c9335..e2f2e1e 100644
--- a/config/StdGNU.mk
+++ b/config/StdGNU.mk
@@ -67,7 +67,6 @@ XEN_CONFIG_DIR = $(CONFIG_DIR)/xen
 XEN_SCRIPT_DIR = $(XEN_CONFIG_DIR)/scripts
 
 SOCKET_LIBS =
-PTHREAD_LIBS = -lpthread
 UTIL_LIBS = -lutil
 DLOPEN_LIBS = -ldl
 
diff --git a/config/Tools.mk.in b/config/Tools.mk.in
index 339a7b6..912d021 100644
--- a/config/Tools.mk.in
+++ b/config/Tools.mk.in
@@ -26,6 +26,10 @@ PREPEND_LIB         := @PREPEND_LIB@
 APPEND_INCLUDES     := @APPEND_INCLUDES@
 APPEND_LIB          := @APPEND_LIB@
 
+PTHREAD_CFLAGS      := @PTHREAD_CFLAGS@
+PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
+PTHREAD_LIBS        := @PTHREAD_LIBS@
+
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
 # may block it). We make it the default, but if your GIT repository downloads
diff --git a/tools/configure b/tools/configure
index 64b7eb5..86618f5 100755
--- a/tools/configure
+++ b/tools/configure
@@ -602,6 +602,9 @@ POW_LIB
 LIBOBJS
 ALLOCA
 libiconv
+PTHREAD_LIBS
+PTHREAD_LDFLAGS
+PTHREAD_CFLAGS
 libgcrypt
 libext2fs
 system_aio
@@ -3861,6 +3864,9 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
 
 
 
+
+
+
 # pkg.m4 - Macros to locate and utilise pkg-config.            -*- Autoconf -*-
 # serial 1 (pkg-config-0.24)
 #
@@ -3924,6 +3930,22 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
 
 
 
+# We define, separately, PTHREAD_CFLAGS, _LDFLAGS and _LIBS
+# even though currently we don't set them very separately.
+# This means that the makefiles will not need to change in
+# the future if we make the test more sophisticated.
+
+
+
+# We invoke AX_PTHREAD_VARS with the name of another macro
+# which is then expanded once for each variable.
+
+
+
+
+
+
+
 
 # Enable/disable options
 
@@ -7228,47 +7250,70 @@ else
 fi
 
 
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lpthread" >&5
-$as_echo_n "checking for pthread_create in -lpthread... " >&6; }
-if test "${ac_cv_lib_pthread_pthread_create+set}" = set; then :
+
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread flag" >&5
+$as_echo_n "checking for pthread flag... " >&6; }
+if test "${ax_cv_pthread_flags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
-  ac_check_lib_save_LIBS=$LIBS
-LIBS="-lpthread  $LIBS"
-cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+
+        ax_cv_pthread_flags=-pthread
+
+    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LIBS=""
+
+
+    saved_CFLAGS="$CFLAGS"
+
+    saved_LDFLAGS="$LDFLAGS"
+
+    saved_LIBS="$LIBS"
+
+
+    CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
+
+    LDFLAGS="$LDFLAGS $PTHREAD_LDFLAGS"
+
+    LIBS="$LIBS $PTHREAD_LIBS"
+
+        cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 
-/* Override any GCC internal prototype to avoid an error.
-   Use char because int might match the return type of a GCC
-   builtin and then its argument prototype would still apply.  */
-#ifdef __cplusplus
-extern "C"
-#endif
-char pthread_create ();
-int
-main ()
-{
-return pthread_create ();
-  ;
-  return 0;
+#include <pthread.h>
+int main(void) {
+  pthread_atfork(0,0,0);
+  pthread_create(0,0,0,0);
 }
+
 _ACEOF
 if ac_fn_c_try_link "$LINENO"; then :
-  ac_cv_lib_pthread_pthread_create=yes
+
 else
-  ac_cv_lib_pthread_pthread_create=no
+  ax_cv_pthread_flags=failed
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext conftest.$ac_ext
-LIBS=$ac_check_lib_save_LIBS
-fi
-{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread_pthread_create" >&5
-$as_echo "$ac_cv_lib_pthread_pthread_create" >&6; }
-if test "x$ac_cv_lib_pthread_pthread_create" = x""yes; then :
 
-else
-  as_fn_error $? "Could not find libpthread" "$LINENO" 5
+    CFLAGS="$saved_CFLAGS"
+
+    LDFLAGS="$saved_LDFLAGS"
+
+    LIBS="$saved_LIBS"
+
+
 fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ax_cv_pthread_flags" >&5
+$as_echo "$ax_cv_pthread_flags" >&6; }
+    if test "x$ax_cv_pthread_flags" = xfailed; then
+        as_fn_error $? "-pthread does not work" "$LINENO" 5
+    fi
+
+    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LIBS=""
+
+
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_gettime in -lrt" >&5
 $as_echo_n "checking for clock_gettime in -lrt... " >&6; }
diff --git a/tools/configure.ac b/tools/configure.ac
index 0204e36..250dffd 100644
--- a/tools/configure.ac
+++ b/tools/configure.ac
@@ -23,6 +23,7 @@ AC_USE_SYSTEM_EXTENSIONS
 AC_CANONICAL_HOST
 
 # M4 Macro includes
+m4_include([m4/savevar.m4])
 m4_include([m4/features.m4])
 m4_include([m4/path_or_fail.m4])
 m4_include([m4/python_version.m4])
@@ -33,6 +34,7 @@ m4_include([m4/set_cflags_ldflags.m4])
 m4_include([m4/uuid.m4])
 m4_include([m4/pkg.m4])
 m4_include([m4/curses.m4])
+m4_include([m4/pthread.m4])
 
 # Enable/disable options
 AX_ARG_DEFAULT_DISABLE([githttp], [Download GIT repositories via HTTP])
@@ -129,8 +131,7 @@ AC_CHECK_LIB([ext2fs], [ext2fs_open2], [libext2fs="y"], [libext2fs="n"])
 AC_SUBST(libext2fs)
 AC_CHECK_LIB([gcrypt], [gcry_md_hash_buffer], [libgcrypt="y"], [libgcrypt="n"])
 AC_SUBST(libgcrypt)
-AC_CHECK_LIB([pthread], [pthread_create], [] ,
-    [AC_MSG_ERROR([Could not find libpthread])])
+AX_CHECK_PTHREAD
 AC_CHECK_LIB([rt], [clock_gettime])
 AC_CHECK_LIB([yajl], [yajl_alloc], [],
     [AC_MSG_ERROR([Could not find yajl])])
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index 55eb755..a1ba134 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -73,6 +73,8 @@ CFLAGS   += -I. $(CFLAGS_xeninclude)
 # Needed for posix_fadvise64() in xc_linux.c
 CFLAGS-$(CONFIG_Linux) += -D_GNU_SOURCE
 
+CFLAGS	+= $(PTHREAD_CFLAGS)
+
 # Define this to make it possible to run valgrind on code linked with these
 # libraries.
 #CFLAGS   += -DVALGRIND -O0 -ggdb3
@@ -157,7 +159,7 @@ libxenctrl.so.$(MAJOR): libxenctrl.so.$(MAJOR).$(MINOR)
 	ln -sf $< $@
 
 libxenctrl.so.$(MAJOR).$(MINOR): $(CTRL_PIC_OBJS)
-	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
+	$(CC) $(LDFLAGS) $(PTHREAD_LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
 
 # libxenguest
 
diff --git a/tools/m4/pthread.m4 b/tools/m4/pthread.m4
new file mode 100644
index 0000000..57ea85c
--- /dev/null
+++ b/tools/m4/pthread.m4
@@ -0,0 +1,41 @@
+# We define, separately, PTHREAD_CFLAGS, _LDFLAGS and _LIBS
+# even though currently we don't set them very separately.
+# This means that the makefiles will not need to change in
+# the future if we make the test more sophisticated.
+
+AC_DEFUN([AX_PTHREAD_CV2VARS],[
+    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LIBS=""
+])
+
+# We invoke AX_PTHREAD_VARS with the name of another macro
+# which is then expanded once for each variable.
+AC_DEFUN([AX_PTHREAD_VARS],[$1(CFLAGS) $1(LDFLAGS) $1(LIBS)])
+
+AC_DEFUN([AX_PTHREAD_VAR_APPLY],[
+    $1="$$1 $PTHREAD_$1"
+])
+AC_DEFUN([AX_PTHREAD_VAR_SUBST],[AC_SUBST(PTHREAD_$1)])
+
+AC_DEFUN([AX_CHECK_PTHREAD],[
+    AC_CACHE_CHECK([for pthread flag], [ax_cv_pthread_flags], [
+        ax_cv_pthread_flags=-pthread
+        AX_PTHREAD_CV2VARS
+        AX_PTHREAD_VARS([AX_SAVEVAR_SAVE])
+        AX_PTHREAD_VARS([AX_PTHREAD_VAR_APPLY])
+        AC_LINK_IFELSE([
+#include <pthread.h>
+int main(void) {
+  pthread_atfork(0,0,0);
+  pthread_create(0,0,0,0);
+}
+],[],[ax_cv_pthread_flags=failed])
+        AX_PTHREAD_VARS([AX_SAVEVAR_RESTORE])
+    ])
+    if test "x$ax_cv_pthread_flags" = xfailed; then
+        AC_MSG_ERROR([-pthread does not work])
+    fi
+    AX_PTHREAD_CV2VARS
+    AX_PTHREAD_VARS([AX_PTHREAD_VAR_SUBST])
+])
diff --git a/tools/m4/savevar.m4 b/tools/m4/savevar.m4
new file mode 100644
index 0000000..2156bee
--- /dev/null
+++ b/tools/m4/savevar.m4
@@ -0,0 +1,6 @@
+AC_DEFUN([AX_SAVEVAR_SAVE],[
+    saved_$1="$$1"
+])
+AC_DEFUN([AX_SAVEVAR_RESTORE],[
+    $1="$saved_$1"
+])
-- 
1.7.2.5

[PATCH 08/31] libxl: Use PTHREAD_CFLAGS, LDFLAGS, LIBS

[Ian Jackson]

This is going to be needed for pthread_atfork.  It is a mystery why it
hasn't been needed before.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/Makefile |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 748d057..b0143e6 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -22,6 +22,10 @@ endif
 LIBXL_LIBS =
 LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(UTIL_LIBS) $(LIBUUID_LIBS)
 
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
+LIBXL_LIBS += $(PTHREAD_LIBS)
+
 LIBXLU_LIBS =
 
 LIBXL_OBJS-y = osdeps.o libxl_paths.o libxl_bootloader.o flexarray.o
-- 
1.7.2.5

[PATCH 09/31] tools: Use PTHREAD_CFLAGS, _LDFLAGS, _LIBS

[Ian Jackson]

Replace all literal occurrences of -lpthread and -pthread in Makefiles
by references to PTHREAD_CFLAGS, PTHREAD_LDFLAGS and PTHREAD_LIBS.
These are the new variables set by configure, and currently expand to
-pthread on the compilation and link lines as is required.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/blktap/drivers/Makefile       |    7 +++++--
 tools/libfsimage/common/Makefile    |    5 ++++-
 tools/vtpm_manager/manager/Makefile |    4 +++-
 tools/xenpaging/Makefile            |    5 +++--
 4 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/tools/blktap/drivers/Makefile b/tools/blktap/drivers/Makefile
index addb2fe..941592e 100644
--- a/tools/blktap/drivers/Makefile
+++ b/tools/blktap/drivers/Makefile
@@ -35,8 +35,11 @@ else
 AIOLIBS     := -laio
 endif
 
-LDLIBS_blktapctrl := $(MEMSHRLIBS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -L../lib -lblktap -lrt -lm -lpthread
-LDLIBS_img := $(AIOLIBS) $(CRYPT_LIB) -lpthread -lz
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
+
+LDLIBS_blktapctrl := $(MEMSHRLIBS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -L../lib -lblktap -lrt -lm $(PTHREAD_LIBS)
+LDLIBS_img := $(AIOLIBS) $(CRYPT_LIB) $(PTHREAD_LIBS) -lz
 
 BLK-OBJS-y  := block-aio.o
 BLK-OBJS-y  += block-sync.o
diff --git a/tools/libfsimage/common/Makefile b/tools/libfsimage/common/Makefile
index 11621e7..3684913 100644
--- a/tools/libfsimage/common/Makefile
+++ b/tools/libfsimage/common/Makefile
@@ -8,6 +8,9 @@ LDFLAGS-$(CONFIG_SunOS) = -Wl,-M -Wl,mapfile-SunOS
 LDFLAGS-$(CONFIG_Linux) = -Wl,mapfile-GNU
 LDFLAGS = $(LDFLAGS-y)
 
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
+
 LIB_SRCS-y = fsimage.c fsimage_plugin.c fsimage_grub.c
 
 PIC_OBJS := $(patsubst %.c,%.opic,$(LIB_SRCS-y))
@@ -37,7 +40,7 @@ libfsimage.so.$(MAJOR): libfsimage.so.$(MAJOR).$(MINOR)
 	ln -sf $< $@
 
 libfsimage.so.$(MAJOR).$(MINOR): $(PIC_OBJS)
-	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libfsimage.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ -lpthread
+	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libfsimage.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(PTHREAD_LIBS)
 
 -include $(DEPS)
 
diff --git a/tools/vtpm_manager/manager/Makefile b/tools/vtpm_manager/manager/Makefile
index 149f01d..7564af1 100644
--- a/tools/vtpm_manager/manager/Makefile
+++ b/tools/vtpm_manager/manager/Makefile
@@ -33,4 +33,6 @@ $(BIN): $(OBJS)
 
 # libraries
 LIBS += ../tcs/libTCS.a ../util/libTCGUtils.a ../crypto/libtcpaCrypto.a
-LIBS += -lcrypto -lpthread -lm
+LIBS += -lcrypto $(PTHREAD_LIBS) -lm
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
diff --git a/tools/xenpaging/Makefile b/tools/xenpaging/Makefile
index 08230a6..548d9dd 100644
--- a/tools/xenpaging/Makefile
+++ b/tools/xenpaging/Makefile
@@ -1,8 +1,9 @@
 XEN_ROOT=$(CURDIR)/../..
 include $(XEN_ROOT)/tools/Rules.mk
 
-CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenstore)
-LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -pthread
+CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenstore) $(PTHREAD_CFLAGS)
+LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(PTHREAD_LIBS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
 
 POLICY    = default
 
-- 
1.7.2.5

[PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Jackson]

Formally change the libxl memory allocation failure policy to "crash".

Previously we had a very uneven approach; much code assumed that
libxl__sprintf (for example) would never return NULL, but some code
was written more carefully.

We think it is unlikely that we will be able to make the library
actually robust against allocation failure (since that would be an
awful lot of never-tested error paths) and few calling environments
will be able to cope anyway.  So, instead, adopt the alternative
approach: provide allocation functions which never return null, but
will crash the whole process instead.

Consequently,
 - New noreturn function libxl__alloc_failed which may be used for
   printing a vaguely-useful error message, rather than simply
   dereferencing a null pointer.
 - libxl__ptr_add now returns void as it crashes on failure.
 - libxl__zalloc, _calloc, _strdup, _strndup, crash on failure using
   libxl__alloc_failed.  So all the code that uses these can no longer
   dereference null on malloc failure.

While we're at it, make libxl__ptr_add use realloc rather than
emulating it with calloc and free, and make it grow the array
exponentially rather than linearly.

Things left to do:
 - Provide versions of malloc, realloc and free which do the
   checking but which do not enroll the pointer in the gc.
 - Remove a lot of now-spurious error handling.
 - Remove the ERROR_NOMEM error code.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.h          |    4 ++
 tools/libxl/libxl_internal.c |   87 ++++++++++++++++++++---------------------
 tools/libxl/libxl_internal.h |    8 +++-
 3 files changed, 53 insertions(+), 46 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 2aec910..0219f81 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -194,6 +194,10 @@
  * No temporary objects allocated from the pool may be explicitly freed.
  * Therefore public functions which initialize a libxl__gc MUST call
  * libxl__free_all() before returning.
+ *
+ * Memory allocation failures are not handled gracefully.  If malloc
+ * (or realloc) fails, libxl will cause the entire process to print
+ * a message to stderr and exit with status 255.
  */
 /*
  * libxl types
diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
index 12c32dc..2270234 100644
--- a/tools/libxl/libxl_internal.c
+++ b/tools/libxl/libxl_internal.c
@@ -17,40 +17,45 @@
 
 #include "libxl_internal.h"
 
-int libxl__error_set(libxl__gc *gc, int code)
-{
-    return 0;
+void libxl__alloc_failed(libxl_ctx *ctx, const char *func,
+                         size_t nmemb, size_t size) {
+#define M "libxl: FATAL ERROR: memory allocation failure"
+#define L (size ? M " (%s, %lu x %lu)\n" : M " (%s)\n"), \
+          func, (unsigned long)nmemb, (unsigned long)size
+    libxl__log(ctx, XTL_CRITICAL, ENOMEM, 0,0, func, L);
+    fprintf(stderr, L);
+    fflush(stderr);
+    _exit(-1);
+#undef M
+#undef L
 }
 
-int libxl__ptr_add(libxl__gc *gc, void *ptr)
+void libxl__ptr_add(libxl__gc *gc, void *ptr)
 {
     int i;
-    void **re;
 
     if (!ptr)
-        return 0;
+        return;
 
     /* fast case: we have space in the array for storing the pointer */
     for (i = 0; i < gc->alloc_maxsize; i++) {
         if (!gc->alloc_ptrs[i]) {
             gc->alloc_ptrs[i] = ptr;
-            return 0;
+            return;
         }
     }
-    /* realloc alloc_ptrs manually with calloc/free/replace */
-    re = calloc(gc->alloc_maxsize + 25, sizeof(void *));
-    if (!re)
-        return -1;
-    for (i = 0; i < gc->alloc_maxsize; i++)
-        re[i] = gc->alloc_ptrs[i];
-    /* assign the next pointer */
-    re[i] = ptr;
+    int new_maxsize = gc->alloc_maxsize * 2 + 25;
+    assert(new_maxsize < INT_MAX / sizeof(void*) / 2);
+    gc->alloc_ptrs = realloc(gc->alloc_ptrs, new_maxsize * sizeof(void *));
+    if (!gc->alloc_ptrs)
+        libxl__alloc_failed(CTX, __func__, sizeof(void*), new_maxsize);
 
-    /* replace the old alloc_ptr */
-    free(gc->alloc_ptrs);
-    gc->alloc_ptrs = re;
-    gc->alloc_maxsize += 25;
-    return 0;
+    gc->alloc_ptrs[gc->alloc_maxsize++] = ptr;
+
+    while (gc->alloc_maxsize < new_maxsize)
+        gc->alloc_ptrs[gc->alloc_maxsize++] = 0;
+
+    return;
 }
 
 void libxl__free_all(libxl__gc *gc)
@@ -71,10 +76,7 @@ void libxl__free_all(libxl__gc *gc)
 void *libxl__zalloc(libxl__gc *gc, int bytes)
 {
     void *ptr = calloc(bytes, 1);
-    if (!ptr) {
-        libxl__error_set(gc, ENOMEM);
-        return NULL;
-    }
+    if (!ptr) libxl__alloc_failed(CTX, __func__, bytes, 1);
 
     libxl__ptr_add(gc, ptr);
     return ptr;
@@ -83,10 +85,7 @@ void *libxl__zalloc(libxl__gc *gc, int bytes)
 void *libxl__calloc(libxl__gc *gc, size_t nmemb, size_t size)
 {
     void *ptr = calloc(nmemb, size);
-    if (!ptr) {
-        libxl__error_set(gc, ENOMEM);
-        return NULL;
-    }
+    if (!ptr) libxl__alloc_failed(CTX, __func__, nmemb, size);
 
     libxl__ptr_add(gc, ptr);
     return ptr;
@@ -97,9 +96,8 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size)
     void *new_ptr = realloc(ptr, new_size);
     int i = 0;
 
-    if (new_ptr == NULL && new_size != 0) {
-        return NULL;
-    }
+    if (new_ptr == NULL && new_size != 0)
+        libxl__alloc_failed(CTX, __func__, new_size, 1);
 
     if (ptr == NULL) {
         libxl__ptr_add(gc, new_ptr);
@@ -112,7 +110,6 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size)
         }
     }
 
-
     return new_ptr;
 }
 
@@ -126,16 +123,13 @@ char *libxl__sprintf(libxl__gc *gc, const char *fmt, ...)
     ret = vsnprintf(NULL, 0, fmt, ap);
     va_end(ap);
 
-    if (ret < 0) {
-        return NULL;
-    }
+    assert(ret >= 0);
 
     s = libxl__zalloc(gc, ret + 1);
-    if (s) {
-        va_start(ap, fmt);
-        ret = vsnprintf(s, ret + 1, fmt, ap);
-        va_end(ap);
-    }
+    va_start(ap, fmt);
+    ret = vsnprintf(s, ret + 1, fmt, ap);
+    va_end(ap);
+
     return s;
 }
 
@@ -143,8 +137,9 @@ char *libxl__strdup(libxl__gc *gc, const char *c)
 {
     char *s = strdup(c);
 
-    if (s)
-        libxl__ptr_add(gc, s);
+    if (!s) libxl__alloc_failed(CTX, __func__, strlen(c), 1);
+
+    libxl__ptr_add(gc, s);
 
     return s;
 }
@@ -153,8 +148,7 @@ char *libxl__strndup(libxl__gc *gc, const char *c, size_t n)
 {
     char *s = strndup(c, n);
 
-    if (s)
-        libxl__ptr_add(gc, s);
+    if (!s) libxl__alloc_failed(CTX, __func__, n, 1);
 
     return s;
 }
@@ -175,6 +169,9 @@ void libxl__logv(libxl_ctx *ctx, xentoollog_level msglevel, int errnoval,
              const char *file, int line, const char *func,
              const char *fmt, va_list ap)
 {
+    /* WARNING this function may not call any libxl-provided
+     * memory allocation function, as those may
+     * call libxl__alloc_failed which calls libxl__logv. */
     char *enomem = "[out of memory formatting log message]";
     char *base = NULL;
     int rc, esave;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 04f13f6..2ad446a 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -116,6 +116,12 @@ typedef struct libxl__gc libxl__gc;
 typedef struct libxl__egc libxl__egc;
 typedef struct libxl__ao libxl__ao;
 
+void libxl__alloc_failed(libxl_ctx *, const char *func,
+                         size_t nmemb, size_t size) __attribute__((noreturn));
+  /* func, size and nmemb are used only in the log message.
+   * You may pass size==0 if size and nmemb are not meaningful
+   * and should not be printed. */
+
 typedef struct libxl__ev_fd libxl__ev_fd;
 typedef void libxl__ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
                                    int fd, short events, short revents);
@@ -380,7 +386,7 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
  * collection on exit from the outermost libxl callframe.
  */
 /* register @ptr in @gc for free on exit from outermost libxl callframe. */
-_hidden int libxl__ptr_add(libxl__gc *gc, void *ptr);
+_hidden void libxl__ptr_add(libxl__gc *gc, void *ptr);
 /* if this is the outermost libxl callframe then free all pointers in @gc */
 _hidden void libxl__free_all(libxl__gc *gc);
 /* allocate and zero @bytes. (similar to a gc'd malloc(3)+memzero()) */
-- 
1.7.2.5

[PATCH 11/31] libxl: Make libxl__zalloc et al tolerate a NULL gc

[Ian Jackson]

Arrange that if we pass NULL as a gc, we simply don't register the
pointer.  This instantly gives us non-gc'ing but error-checking
versions of malloc, realloc, vasprintf, etc.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl_internal.c |    5 ++++-
 tools/libxl/libxl_internal.h |   21 +++++++++++++--------
 2 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
index 2270234..541f20b 100644
--- a/tools/libxl/libxl_internal.c
+++ b/tools/libxl/libxl_internal.c
@@ -34,6 +34,9 @@ void libxl__ptr_add(libxl__gc *gc, void *ptr)
 {
     int i;
 
+    if (!gc)
+        return;
+
     if (!ptr)
         return;
 
@@ -101,7 +104,7 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size)
 
     if (ptr == NULL) {
         libxl__ptr_add(gc, new_ptr);
-    } else if (new_ptr != ptr) {
+    } else if (new_ptr != ptr && gc != NULL) {
         for (i = 0; i < gc->alloc_maxsize; i++) {
             if (gc->alloc_ptrs[i] == ptr) {
                 gc->alloc_ptrs[i] = new_ptr;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 2ad446a..9340bde 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -384,30 +384,35 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
  *
  * All pointers returned by these functions are registered for garbage
  * collection on exit from the outermost libxl callframe.
+ *
+ * However, where the argument is stated to be "gc_opt", NULL may be
+ * passed instead, in which case no garbage collection will occur; the
+ * pointer must later be freed with free().  This is for memory
+ * allocations of types (b) and (c).
  */
 /* register @ptr in @gc for free on exit from outermost libxl callframe. */
-_hidden void libxl__ptr_add(libxl__gc *gc, void *ptr);
+_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr);
 /* if this is the outermost libxl callframe then free all pointers in @gc */
 _hidden void libxl__free_all(libxl__gc *gc);
 /* allocate and zero @bytes. (similar to a gc'd malloc(3)+memzero()) */
-_hidden void *libxl__zalloc(libxl__gc *gc, int bytes);
+_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes);
 /* allocate and zero memory for an array of @nmemb members of @size each.
  * (similar to a gc'd calloc(3)). */
-_hidden void *libxl__calloc(libxl__gc *gc, size_t nmemb, size_t size);
+_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size);
 /* change the size of the memory block pointed to by @ptr to @new_size bytes.
  * unlike other allocation functions here any additional space between the
  * oldsize and @new_size is not initialised (similar to a gc'd realloc(3)). */
-_hidden void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size);
+_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size);
 /* print @fmt into an allocated string large enoughto contain the result.
  * (similar to gc'd asprintf(3)). */
-_hidden char *libxl__sprintf(libxl__gc *gc, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3);
+_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3);
 /* duplicate the string @c (similar to a gc'd strdup(3)). */
-_hidden char *libxl__strdup(libxl__gc *gc, const char *c);
+_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c);
 /* duplicate at most @n bytes of string @c (similar to a gc'd strndup(3)). */
-_hidden char *libxl__strndup(libxl__gc *gc, const char *c, size_t n);
+_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n);
 /* strip the last path component from @s and return as a newly allocated
  * string. (similar to a gc'd dirname(3)). */
-_hidden char *libxl__dirname(libxl__gc *gc, const char *s);
+_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s);
 
 _hidden char **libxl__xs_kvs_of_flexarray(libxl__gc *gc, flexarray_t *array, int length);
 
-- 
1.7.2.5

[PATCH 12/31] libxl: Introduce some convenience macros

[Ian Jackson]

We introduce:
   <type> *GCNEW(<type> *var);
   <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
   <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
   char *GCSPRINTF(const char *fmt, ...);
   void LOG(<xtl_level_suffix>, const char *fmt, ...);
   void LOGE(<xtl_level_suffix>, const char *fmt, ...);
   void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
all of which expect, in the calling context,
   libxl__gc *gc;

Most of these will find callers in subsequent patches.  The exceptions
are the orthogonally necessary LOGE and LOGEV, and GCREALLOC_ARRAY.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl_internal.h |   72 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 72 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9340bde..95c34f3 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1349,6 +1349,78 @@ _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
 #define GC_FREE       libxl__free_all(gc)
 #define CTX           libxl__gc_owner(gc)
 
+/* Allocation macros all of which use the gc. */
+
+#define ARRAY_SIZE_OK(ptr, nmemb) ((nmemb) < INT_MAX / (sizeof(*(ptr)) * 2))
+
+/*
+ * Expression statement  <type> *GCNEW(<type> *var);
+ * Uses                  libxl__gc *gc;
+ *
+ * Allocates a new object of type <type> from the gc and zeroes it
+ * with memset.  Sets var to point to the new object or zero (setting
+ * errno).  Returns the new value of var.
+ */
+#define GCNEW(var)                                      \
+    (((var) = libxl__zalloc((gc),sizeof(*(var)))))
+
+/*
+ * Expression statement  <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
+ * Uses                  libxl__gc *gc;
+ *
+ * Like GCNEW but allocates an array of nmemb elements, as if from
+ * calloc.  Does check for integer overflow due to large nmemb.  If
+ * nmemb is 0 may succeed by returning 0.
+ */
+#define GCNEW_ARRAY(var, nmemb)                                 \
+    ((var) = libxl__calloc((gc), (nmemb), sizeof(*(var))))
+    
+/*
+ * Expression statement  <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
+ * Uses                  libxl__gc *gc;
+ *
+ * Reallocates the array var to be of size nmemb elements.  Updates
+ * var and returns the new value of var.  Does check for integer
+ * overflow due to large nmemb.
+ *
+ * Do not pass nmemb==0.  old may be 0 on entry.
+ */
+#define GCREALLOC_ARRAY(var, nmemb)                                     \
+    (assert(nmemb > 0),                                                 \
+     assert(ARRAY_SIZE_OK((var), (nmemb))),                             \
+     (var) = libxl__realloc((gc), (var), (nmemb)*sizeof(*(var)))))
+
+
+/*
+ * Expression            char *GCSPRINTF(const char *fmt, ...);
+ * Uses                  libxl__gc *gc;
+ *
+ * Trivial convenience wrapper for libxl__sprintf.
+ */
+#define GCSPRINTF(fmt, ...) (libxl__sprintf((gc), (fmt), __VA_ARGS__))
+
+
+/*
+ * Expression statements
+ *    void LOG(<xtl_level_suffix>, const char *fmt, ...);
+ *    void LOGE(<xtl_level_suffix>, const char *fmt, ...);
+ *    void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
+ * Use
+ *    libxl__gc *gc;
+ *
+ * Trivial convenience wrappers for LIBXL__LOG, LIBXL__LOG_ERRNO and
+ * LIBXL__LOG_ERRNOVAL, respectively (and thus for libxl__log).
+ *
+ * XTL_<xtl_level_suffix> should exist and be an xentoollog.h log level
+ * So <xtl_level_suffix> should be one of
+ *   DEBUG VERBOSE DETAIL PROGRESS INFO NOTICE WARN ERROR ERROR CRITICAL
+ * Of these, most of libxl uses
+ *   DEBUG INFO WARN ERROR
+ */
+#define LOG(l,f, ...)     LIBXL__LOG(CTX,XTL_##l,(f),##__VA_ARGS__)
+#define LOGE(l,f, ...)    LIBXL__LOG_ERRNO(CTX,XTL_##l,(f),##__VA_ARGS__)
+#define LOGEV(l,e,f, ...) LIBXL__LOG_ERRNOVAL(CTX,XTL_##l,(e),(f),##__VA_ARGS__)
+
 
 /* Locking functions.  See comment for "lock" member of libxl__ctx. */
 
-- 
1.7.2.5

[PATCH 13/31] libxl: include <ctype.h> and introduce CTYPE helper macro

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl_internal.h |   20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 95c34f3..b35421f 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -33,6 +33,7 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <ctype.h>
 
 #include <sys/mman.h>
 #include <sys/poll.h>
@@ -1469,6 +1470,25 @@ static inline void libxl__ctx_unlock(libxl_ctx *ctx) {
     } while(0)
 
 
+/*
+ * int CTYPE(ISFOO, char c);
+ * int CTYPE(toupper, char c);
+ * int CTYPE(tolower, char c);
+ *
+ * This is necessary because passing a simple char to a ctype.h
+ * is forbidden.  ctype.h macros take ints derived from _unsigned_ chars.
+ *
+ * If you have a char which might be EOF then you should already have
+ * it in an int representing an unsigned char, and you can use the
+ * <ctype.h> macros directly.  This generally happens only with values
+ * from fgetc et al.
+ *
+ * For any value known to be a character (eg, anything that came from
+ * a char[]), use CTYPE.
+ */
+#define CTYPE(isfoo,c) (isfoo((unsigned char)(c)))
+
+
 #endif
 
 /*
-- 
1.7.2.5

[PATCH 14/31] libxl: Provide libxl_string_list_length

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl.c |    8 ++++++++
 tools/libxl/libxl.h |    1 +
 2 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index f41b62f..8910420 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -177,6 +177,14 @@ void libxl_string_list_dispose(libxl_string_list *psl)
     free(sl);
 }
 
+int libxl_string_list_length(const libxl_string_list *psl)
+{
+    if (!psl) return 0;
+    int i = 0;
+    while (*psl++) i++;
+    return i;
+}
+
 void libxl_key_value_list_dispose(libxl_key_value_list *pkvl)
 {
     int i;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 0219f81..b376316 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -273,6 +273,7 @@ typedef uint8_t libxl_mac[6];
 
 typedef char **libxl_string_list;
 void libxl_string_list_dispose(libxl_string_list *sl);
+int libxl_string_list_length(const libxl_string_list *sl);
 
 typedef char **libxl_key_value_list;
 void libxl_key_value_list_dispose(libxl_key_value_list *kvl);
-- 
1.7.2.5

[PATCH 15/31] libxl: include <_libxl_paths.h> in libxl_internal.h

[Ian Jackson]

Ie, we permit general code in libxl direct access to the manifest
constants such as XEN_RUN_DIR.  This simplifies their use in (eg)
format strings.

This might be controversial because it will make it difficult to make
any of these runtime-configurable later without changing lots of use
sites.  But I don't think it's likely we'll want to do that.

For the moment, leave existing call sites of all the functions in
libxl_paths.c unchanged.  The simplified use arrangements can be used
in new code and when we update call sites for other reasons.

Also correct the dependencies in the Makefile so that _libxl_paths.h
is generated before anything that uses libxl_internal.h.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 tools/libxl/Makefile         |    4 +---
 tools/libxl/libxl_internal.h |    1 +
 tools/libxl/libxl_paths.c    |    1 -
 3 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index b0143e6..9f7e454 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -102,11 +102,9 @@ _libxl_list.h: $(XEN_INCLUDE)/xen-external/bsd-sys-queue-h-seddery $(XEN_INCLUDE
 	perl $^ --prefix=libxl >$@.new
 	$(call move-if-changed,$@.new,$@)
 
-libxl_paths.c: _libxl_paths.h
-
 libxl.h: _libxl_types.h
 libxl_json.h: _libxl_types_json.h
-libxl_internal.h: _libxl_types_internal.h
+libxl_internal.h: _libxl_types_internal.h _libxl_paths.h
 libxl_internal_json.h: _libxl_types_internal_json.h
 
 $(LIBXL_OBJS) $(LIBXLU_OBJS) $(XL_OBJS): libxl.h
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index b35421f..740bde2 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -51,6 +51,7 @@
 #include <xen/io/xenbus.h>
 
 #include "libxl.h"
+#include "_libxl_paths.h"
 
 #if __GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)
 #define _hidden __attribute__((visibility("hidden")))
diff --git a/tools/libxl/libxl_paths.c b/tools/libxl/libxl_paths.c
index a95d29f..388b344 100644
--- a/tools/libxl/libxl_paths.c
+++ b/tools/libxl/libxl_paths.c
@@ -14,7 +14,6 @@
 
 #include "libxl_osdeps.h" /* must come before any other headers */
 #include "libxl_internal.h"
-#include "_libxl_paths.h"
 
 const char *libxl_sbindir_path(void)
 {
-- 
1.7.2.5

[PATCH 16/31] libxl: abolish libxl_ctx_postfork

[Ian Jackson]

libxl's task has become too complicated (particularly in the presence
of both forking and multithreading) to support reuse of the same
libxl_ctx after fork.

So abolish libxl_ctx_fork.  xl instead simply initialises a new
libxl_ctx.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl.h       |    1 -
 tools/libxl/libxl_utils.c |    7 -------
 tools/libxl/xl.c          |    8 ++++++++
 tools/libxl/xl.h          |    1 +
 tools/libxl/xl_cmdimpl.c  |    8 ++------
 5 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index b376316..edbca53 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -461,7 +461,6 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
                     unsigned flags /* none currently defined */,
                     xentoollog_logger *lg);
 int libxl_ctx_free(libxl_ctx *ctx /* 0 is OK */);
-int libxl_ctx_postfork(libxl_ctx *ctx);
 
 /* domain related functions */
 typedef int (*libxl_console_ready)(libxl_ctx *ctx, uint32_t domid, void *priv);
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index d6cd78d..0cbd85e 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -365,13 +365,6 @@ READ_WRITE_EXACTLY(read, 1, /* */)
 READ_WRITE_EXACTLY(write, 0, const)
 
 
-int libxl_ctx_postfork(libxl_ctx *ctx) {
-    if (ctx->xsh) xs_daemon_destroy_postfork(ctx->xsh);
-    ctx->xsh = xs_daemon_open();
-    if (!ctx->xsh) return ERROR_FAIL;
-    return 0;
-}
-
 pid_t libxl_fork(libxl_ctx *ctx)
 {
     pid_t pid;
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index 2b14814..62c0abd 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -94,6 +94,14 @@ static void parse_global_config(const char *configfile,
     xlu_cfg_destroy(config);
 }
 
+void postfork(void)
+{
+    if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
+        fprintf(stderr, "cannot reinit xl context after fork\n");
+        exit(-1);
+    }
+}
+
 int main(int argc, char **argv)
 {
     int opt = 0;
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index 0a3d628..7e258d5 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -105,6 +105,7 @@ struct cmd_spec *cmdtable_lookup(const char *s);
 
 extern libxl_ctx *ctx;
 extern xentoollog_logger_stdiostream *logger;
+void postfork(void);
 
 /* global options */
 extern int autoballoon;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 6f4dd09..c9e9943 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1441,7 +1441,7 @@ static int autoconnect_console(libxl_ctx *ctx, uint32_t domid, void *priv)
     } else if (*pid > 0)
         return 0;
 
-    libxl_ctx_postfork(ctx);
+    postfork();
 
     sleep(1);
     libxl_primary_console_exec(ctx, domid);
@@ -1728,11 +1728,7 @@ start:
             goto out;
         }
 
-        rc = libxl_ctx_postfork(ctx);
-        if (rc) {
-            LOG("failed to reinitialise context after fork");
-            exit(-1);
-        }
+        postfork();
 
         if (asprintf(&name, "xl-%s", d_config.c_info.name) < 0) {
             LOG("Failed to allocate memory in asprintf");
-- 
1.7.2.5

[PATCH 17/31] libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS

[Ian Jackson]

Introduce definition and use of a new function-local macro REQUIRE_FDS
to avoid repeatedly spelling out which fds we are interested in.

We are going to introduce a new fd for the SIGCHLD self-pipe.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl_event.c |   82 ++++++++++++++++++++++++++++++--------------
 1 files changed, 56 insertions(+), 26 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 9cb172a..638b9ab 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -593,6 +593,45 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
     int rc;
 
     /*
+     * We need to look at the fds we want twice: firstly, to count
+     * them so we can make the rindex array big enough, and secondly
+     * to actually fill the arrays in.
+     *
+     * To ensure correctness and avoid repeating the logic for
+     * deciding which fds are relevant, we define a macro
+     *    REQUIRE_FDS( BODY )
+     * which calls
+     *    do{
+     *        int req_fd;
+     *        int req_events;
+     *        BODY;
+     *    }while(0)
+     * for each fd with a nonzero events.  This is invoked twice.
+     *
+     * The definition of REQUIRE_FDS is simplified with the helper
+     * macro
+     *    void REQUIRE_FD(int req_fd, int req_events, BODY);
+     */
+
+#define REQUIRE_FDS(BODY) do{                                          \
+                                                                       \
+        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)                     \
+            REQUIRE_FD(efd->fd, efd->events, BODY);                    \
+                                                                       \
+        REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
+                                                                       \
+    }while(0)
+
+#define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
+        int req_events = (req_events_);                 \
+        int req_fd = (req_fd_);                         \
+        if (req_events) {                               \
+            BODY;                                       \
+        }                                               \
+    }while(0)
+
+
+    /*
      * In order to be able to efficiently find the libxl__ev_fd
      * for a struct poll during _afterpoll, we maintain a shadow
      * data structure in CTX->fd_beforepolled: each slot in
@@ -609,13 +648,13 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
          * not to mess with fd_rindex.
          */
 
-        int maxfd = poller->wakeup_pipe[0] + 1;
-        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
-            if (!efd->events)
-                continue;
-            if (efd->fd >= maxfd)
-                maxfd = efd->fd + 1;
-        }
+        int maxfd = 0;
+
+        REQUIRE_FDS({
+            if (req_fd >= maxfd)
+                maxfd = req_fd + 1;
+        });
+
         /* make sure our array is as big as *nfds_io */
         if (poller->fd_rindex_allocd < maxfd) {
             assert(maxfd < INT_MAX / sizeof(int) / 2);
@@ -630,25 +669,16 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
 
     int used = 0;
 
-#define REQUIRE_FD(req_fd, req_events, efd) do{                 \
-        if ((req_events)) {                                     \
-            if (used < *nfds_io) {                              \
-                fds[used].fd = (req_fd);                        \
-                fds[used].events = (req_events);                \
-                fds[used].revents = 0;                          \
-                assert((req_fd) < poller->fd_rindex_allocd);    \
-                poller->fd_rindex[(req_fd)] = used;             \
-            }                                                   \
-            used++;                                             \
-        }                                                       \
-    }while(0)
-
-    LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)
-        REQUIRE_FD(efd->fd, efd->events, efd);
-
-    REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, 0);
-
-#undef REQUIRE_FD
+    REQUIRE_FDS({
+        if (used < *nfds_io) {
+            fds[used].fd = req_fd;
+            fds[used].events = req_events;
+            fds[used].revents = 0;
+            assert(req_fd < poller->fd_rindex_allocd);
+            poller->fd_rindex[req_fd] = used;
+        }
+        used++;
+    });
 
     rc = used <= *nfds_io ? 0 : ERROR_BUFFERFULL;
 
-- 
1.7.2.5

[PATCH 18/31] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Jackson]

We introduce a new "carefd" concept, which relates to fds that we care
about not being inherited by long-lived children.

As yet we do not use this anywhere in libxl.  Until all locations in
libxl which make such fds are converted, libxl__postfork may not work
entirely properly.  If these locations do not use O_CLOEXEC (or use
calls for which there is no O_CLOEXEC) then multithreaded programs may
not work properly.

This introduces a new API call libxl_postfork_child_noexec which must
be called by applications which make long-running non-execing
children.  Add the appropriate call to xl's postfork function.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/Makefile         |    2 +-
 tools/libxl/libxl.c          |    3 +
 tools/libxl/libxl_event.h    |   13 ++++
 tools/libxl/libxl_fork.c     |  146 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   63 ++++++++++++++++++
 tools/libxl/xl.c             |    3 +
 6 files changed, 229 insertions(+), 1 deletions(-)
 create mode 100644 tools/libxl/libxl_fork.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 9f7e454..e5ea867 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -53,7 +53,7 @@ LIBXL_LIBS += -lyajl
 LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
 			libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
 			libxl_internal.o libxl_utils.o libxl_uuid.o libxl_json.o \
-			libxl_qmp.o libxl_event.o $(LIBXL_OBJS-y)
+			libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
 LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
 
 $(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include $(XEN_ROOT)/tools/config.h
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 8910420..60dbfdc 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -68,6 +68,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
 
     /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
 
+    rc = libxl__atfork_init(ctx);
+    if (rc) goto out;
+
     rc = libxl__poller_init(ctx, &ctx->poller_app);
     if (rc) goto out;
 
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index ea553f6..2d2196f 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -371,6 +371,19 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
  */
 void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 
+
+/*
+ * An application which initialises a libxl_ctx in a parent process
+ * and then forks a child which does not quickly exec, must
+ * instead libxl_postfork_child_noexec in the child.  One call
+ * on any existing (or specially made) ctx is sufficient; after
+ * this all previously existing libxl_ctx's are invalidated and
+ * must not be used - or even freed.  It is harmless to call this
+ * postfork function and then exec anyway.
+ */
+void libxl_postfork_child_noexec(libxl_ctx *ctx);
+
+
 #endif
 
 /*
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
new file mode 100644
index 0000000..4751ef4
--- /dev/null
+++ b/tools/libxl/libxl_fork.c
@@ -0,0 +1,146 @@
+/*
+ * Copyright (C) 2012      Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+/*
+ * Internal child process machinery for use by other parts of libxl
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*
+ * carefd arrangements
+ *
+ * carefd_begin and _unlock take out the no_forking lock, which we
+ * also take and release in our pthread_atfork handlers.  So when this
+ * lock is held the whole process cannot fork.  We therefore protect
+ * our fds from leaking into children made by other threads.
+ *
+ * We maintain a list of all the carefds, so that if the application
+ * wants to fork a long-running but non-execing child, we can close
+ * them all.
+ *
+ * So the record function sets CLOEXEC for the benefit of execing
+ * children, and makes a note of the fd for the benefit of non-execing
+ * ones.
+ */
+
+struct libxl__carefd {
+    LIBXL_LIST_ENTRY(libxl__carefd) entry;
+    int fd;
+};
+
+static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;
+static int atfork_registered;
+static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
+    LIBXL_LIST_HEAD_INITIALIZER(carefds);
+
+static void atfork_lock(void)
+{
+    int r = pthread_mutex_lock(&no_forking);
+    assert(!r);
+}
+
+static void atfork_unlock(void)
+{
+    int r = pthread_mutex_unlock(&no_forking);
+    assert(!r);
+}
+
+int libxl__atfork_init(libxl_ctx *ctx)
+{
+    int r, rc;
+    
+    atfork_lock();
+    if (atfork_registered) { rc = 0; goto out; }
+
+    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
+    if (r) libxl__alloc_failed(ctx, __func__, 0,0);
+
+    atfork_registered = 1;
+    rc = 0;
+ out:
+    atfork_unlock();
+    return rc;
+}
+
+void libxl__carefd_begin(void) { atfork_lock(); }
+void libxl__carefd_unlock(void) { atfork_unlock(); }
+
+libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
+{
+    libxl__carefd *cf = 0;
+
+    libxl_fd_set_cloexec(ctx, fd, 1);
+    cf = libxl__zalloc(NULL, sizeof(*cf));
+    cf->fd = fd;
+    LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry);
+    return cf;
+}
+
+libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd)
+{
+    libxl__carefd *cf = 0;
+
+    cf = libxl__carefd_record(ctx, fd);
+    libxl__carefd_unlock();
+    return cf;
+}
+
+void libxl_postfork_child_noexec(libxl_ctx *ctx)
+{
+    libxl__carefd *cf, *cf_tmp;
+    int r;
+
+    atfork_lock();
+    LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
+        if (cf->fd >= 0) {
+            r = close(cf->fd);
+            if (r)
+                LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING,
+                                 "failed to close fd=%d"
+                                 " (perhaps of another libxl ctx)", cf->fd);
+        }
+        free(cf);
+    }
+    LIBXL_LIST_INIT(&carefds);
+    atfork_unlock();
+}
+
+int libxl__carefd_close(libxl__carefd *cf)
+{
+    if (!cf) return 0;
+    int r = cf->fd < 0 ? 0 : close(cf->fd);
+    int esave = errno;
+    atfork_lock();
+    LIBXL_LIST_REMOVE(cf, entry);
+    atfork_unlock();
+    free(cf);
+    errno = esave;
+    return r;
+}
+
+int libxl__carefd_fd(const libxl__carefd *cf)
+{
+    if (!cf) return -1;
+    return cf->fd;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 740bde2..a8372bb 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -611,6 +611,9 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
 
 
+int libxl__atfork_init(libxl_ctx *ctx);
+
+
 /* from xl_dom */
 _hidden libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid);
 _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
@@ -1307,6 +1310,66 @@ _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
 /* For use by ao machinery ONLY */
 _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
 
+
+/*
+ * File descriptors and CLOEXEC
+ */
+
+/*
+ * For libxl functions which create file descriptors, at least one
+ * of the following must be true:
+ *  (a) libxl does not care if copies of this open-file are inherited
+ *      by random children and might remain open indefinitely
+ *  (b) libxl must take extra care for the fd (the actual descriptor,
+ *      not the open-file) as below.  We call this a "carefd".
+ *
+ * The rules for opening a carefd are:
+ *  (i)   Before bringing any carefds into existence,
+ *        libxl code must call libxl__carefd_begin.
+ *  (ii)  Then for each carefd brought into existence,
+ *        libxl code must call libxl__carefd_record
+ *        and remember the libxl__carefd_record*.
+ *  (iii) Then it must call libxl__carefd_unlock.
+ *  (iv)  When in a child process the fd is to be passed across
+ *        exec by libxl, the libxl code must unset FD_CLOEXEC
+ *        on the fd eg by using libxl_fd_set_cloexec.
+ *  (v)   Later, when the fd is to be closed in the same process,
+ *        libxl code must not call close.  Instead, it must call
+ *        libxl__carefd_close.
+ * Steps (ii) and (iii) can be combined by calling the convenience
+ * function libxl__carefd_opened.
+ */
+/* libxl__carefd_begin and _unlock (or _opened) must be called always
+ * in pairs.  They may be called with the CTX lock held.  In between
+ * _begin and _unlock, the following are prohibited:
+ *   - anything which might block
+ *   - any callbacks to the application
+ *   - nested calls to libxl__carefd_begin
+ *   - fork (libxl__fork)
+ * In general nothing should be done before _unlock that could be done
+ * afterwards.
+ */
+typedef struct libxl__carefd libxl__carefd;
+
+void libxl__carefd_begin(void);
+void libxl__carefd_unlock(void);
+
+/* fd may be -1, in which case this returns a dummy libxl__fd_record
+ * on which it _carefd_close is a no-op.  Cannot fail. */
+libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd);
+
+/* Combines _record and _unlock in a single call.  If fd==-1,
+ * still does the unlock, but returns 0.  Cannot fail. */
+libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd);
+
+/* Works just like close(2).  You may pass NULL, in which case it's
+ * a successful no-op. */
+int libxl__carefd_close(libxl__carefd*);
+
+/* You may pass NULL in which case the answer is -1. */
+int libxl__carefd_fd(const libxl__carefd*);
+
+
 /*
  * Convenience macros.
  */
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index 62c0abd..a6ffd25 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -96,6 +96,9 @@ static void parse_global_config(const char *configfile,
 
 void postfork(void)
 {
+    libxl_postfork_child_noexec(ctx); /* in case we don't exit/exec */
+    ctx = 0;
+
     if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
         fprintf(stderr, "cannot reinit xl context after fork\n");
         exit(-1);
-- 
1.7.2.5

[PATCH 19/31] libxl: provide STATE_AO_GC

[Ian Jackson]

Provide a convenience macro for use in ao callback functions, and
document that it should be used.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_internal.h |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index a8372bb..76875bb 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1266,9 +1266,10 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
  * - Note that during callback functions, two gcs are available:
  *    - The one in egc, whose lifetime is only this callback
  *    - The one in ao, whose lifetime is the asynchronous operation
- *   Usually callback function should use CONTAINER_OF
- *   to obtain its own structure, containing a pointer to the ao,
- *   and then use the gc from that ao.
+ *   Usually callback function should use CONTAINER_OF to obtain its
+ *   own state structure, containing a pointer to the ao.  It should
+ *   then obtain the ao and use the ao's gc; this is most easily done
+ *   using the convenience macro STATE_AO_GC.
  */
 
 #define AO_CREATE(ctx, domid, ao_how)                           \
@@ -1298,6 +1299,10 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
 #define AO_GC                                   \
     libxl__gc *const gc = &ao->gc
 
+#define STATE_AO_GC(op)                         \
+    libxl__ao *const ao = op->ao;               \
+    AO_GC
+
 
 /* All of these MUST be called with the ctx locked.
  * libxl__ao_inprogress MUST be called with the ctx locked exactly once. */
-- 
1.7.2.5

[PATCH 20/31] libxl: handle POLLERR, POLLHUP, POLLNVAL properly

[Ian Jackson]

Pass POLLERR and POLLHUP to fd callbacks, as is necessary.
Crash on POLLNVAL since that means our fds are messed up.

Document the behaviour (including the fact that poll sometimes sets
POLLHUP or POLLERR even if only POLLIN was requested.

Fix the one current fd callback to do something with POLLERR|POLLHUP.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_event.c    |    7 ++++++-
 tools/libxl/libxl_internal.h |    5 +++++
 2 files changed, 11 insertions(+), 1 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 638b9ab..5e1a207 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -335,6 +335,9 @@ static void watchfd_callback(libxl__egc *egc, libxl__ev_fd *ev,
 {
     EGC_GC;
 
+    if (revents & (POLLERR|POLLHUP))
+        LIBXL__EVENT_DISASTER(egc, "unexpected poll event on watch fd", 0, 0);
+
     for (;;) {
         char **event = xs_check_watch(CTX->xsh);
         if (!event) {
@@ -739,7 +742,9 @@ static int afterpoll_check_fd(libxl__poller *poller,
         /* again, stale slot entry */
         return 0;
 
-    int revents = fds[slot].revents & events;
+    assert(!(fds[slot].revents & POLLNVAL));
+
+    int revents = fds[slot].revents & (events | POLLERR | POLLHUP);
     /* we mask in case requested events have changed */
 
     return revents;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 76875bb..9f2583d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -127,6 +127,11 @@ void libxl__alloc_failed(libxl_ctx *, const char *func,
 typedef struct libxl__ev_fd libxl__ev_fd;
 typedef void libxl__ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
                                    int fd, short events, short revents);
+  /* Note that revents may contain POLLERR or POLLHUP regardless of
+   * events; otherwise revents contains only bits in events.  Contrary
+   * to the documentation for poll(2), POLLERR and POLLHUP can occur
+   * even if only POLLIN was set in events.  (POLLNVAL is a fatal
+   * error and will cause libxl event machinery to fail an assertion.) */
 struct libxl__ev_fd {
     /* caller should include this in their own struct */
     /* read-only for caller, who may read only when registered: */
-- 
1.7.2.5

[PATCH 21/31] libxl: support multiple libxl__ev_fds for the same fd

[Ian Jackson]

We need a slightly more sophisticated data structure to allow this,
where we record the slot not just for each fd but also for each
(fd,eventbit) where eventbit is POLLIN, POLLPRI, POLLOUT.

Document the new relaxed restriction.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_event.c    |   62 +++++++++++++++++++++++------------------
 tools/libxl/libxl_internal.h |   10 +++++--
 2 files changed, 42 insertions(+), 30 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 5e1a207..672e3fe 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -635,10 +635,11 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
 
 
     /*
-     * In order to be able to efficiently find the libxl__ev_fd
-     * for a struct poll during _afterpoll, we maintain a shadow
-     * data structure in CTX->fd_beforepolled: each slot in
-     * the fds array corresponds to a slot in fd_beforepolled.
+     * In order to be able to efficiently find the libxl__ev_fd for a
+     * struct poll during _afterpoll, we maintain a shadow data
+     * structure in CTX->fd_rindices: each fd corresponds to a slot in
+     * fd_rindices, and each elemebnt in the rindices is three indices
+     * into the fd array (for POLLIN, POLLPRI and POLLOUT).
      */
 
     if (*nfds_io) {
@@ -659,14 +660,16 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
         });
 
         /* make sure our array is as big as *nfds_io */
-        if (poller->fd_rindex_allocd < maxfd) {
-            assert(maxfd < INT_MAX / sizeof(int) / 2);
-            int *newarray = realloc(poller->fd_rindex, sizeof(int) * maxfd);
-            if (!newarray) { rc = ERROR_NOMEM; goto out; }
-            memset(newarray + poller->fd_rindex_allocd, 0,
-                   sizeof(int) * (maxfd - poller->fd_rindex_allocd));
-            poller->fd_rindex = newarray;
-            poller->fd_rindex_allocd = maxfd;
+        if (poller->fd_rindices_allocd < maxfd) {
+            assert(ARRAY_SIZE_OK(poller->fd_rindices, maxfd));
+            poller->fd_rindices =
+                libxl__realloc(0, poller->fd_rindices,
+                               maxfd * sizeof(*poller->fd_rindices));
+            memset(poller->fd_rindices + poller->fd_rindices_allocd,
+                   0,
+                   (maxfd - poller->fd_rindices_allocd)
+                     * sizeof(*poller->fd_rindices));
+            poller->fd_rindices_allocd = maxfd;
         }
     }
 
@@ -677,8 +680,10 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
             fds[used].fd = req_fd;
             fds[used].events = req_events;
             fds[used].revents = 0;
-            assert(req_fd < poller->fd_rindex_allocd);
-            poller->fd_rindex[req_fd] = used;
+            assert(req_fd < poller->fd_rindices_allocd);
+            if (req_events & POLLIN)  poller->fd_rindices[req_fd][0] = used;
+            if (req_events & POLLPRI) poller->fd_rindices[req_fd][1] = used;
+            if (req_events & POLLOUT) poller->fd_rindices[req_fd][2] = used;
         }
         used++;
     });
@@ -706,7 +711,6 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
             *timeout_upd = our_timeout;
     }
 
- out:
     return rc;
 }
 
@@ -728,24 +732,28 @@ static int afterpoll_check_fd(libxl__poller *poller,
                               int fd, int events)
     /* returns mask of events which were requested and occurred */
 {
-    if (fd >= poller->fd_rindex_allocd)
+    if (fd >= poller->fd_rindices_allocd)
         /* added after we went into poll, have to try again */
         return 0;
 
-    int slot = poller->fd_rindex[fd];
+    int i, revents = 0;
+    for (i=0; i<3; i++) {
+        int slot = poller->fd_rindices[fd][i];
 
-    if (slot >= nfds)
-        /* stale slot entry; again, added afterwards */
-        return 0;
+        if (slot >= nfds)
+            /* stale slot entry; again, added afterwards */
+            continue;
 
-    if (fds[slot].fd != fd)
-        /* again, stale slot entry */
-        return 0;
+        if (fds[slot].fd != fd)
+            /* again, stale slot entry */
+            continue;
 
-    assert(!(fds[slot].revents & POLLNVAL));
+        assert(!(fds[slot].revents & POLLNVAL));
+        revents |= fds[slot].revents;
+    }
 
-    int revents = fds[slot].revents & (events | POLLERR | POLLHUP);
     /* we mask in case requested events have changed */
+    revents &= (events | POLLERR | POLLHUP);
 
     return revents;
 }
@@ -1009,7 +1017,7 @@ int libxl__poller_init(libxl_ctx *ctx, libxl__poller *p)
 {
     int r, rc;
     p->fd_polls = 0;
-    p->fd_rindex = 0;
+    p->fd_rindices = 0;
 
     r = pipe(p->wakeup_pipe);
     if (r) {
@@ -1036,7 +1044,7 @@ void libxl__poller_dispose(libxl__poller *p)
     if (p->wakeup_pipe[1] > 0) close(p->wakeup_pipe[1]);
     if (p->wakeup_pipe[0] > 0) close(p->wakeup_pipe[0]);
     free(p->fd_polls);
-    free(p->fd_rindex);
+    free(p->fd_rindices);
 }
 
 libxl__poller *libxl__poller_get(libxl_ctx *ctx)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9f2583d..1a2139c 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -131,7 +131,11 @@ typedef void libxl__ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
    * events; otherwise revents contains only bits in events.  Contrary
    * to the documentation for poll(2), POLLERR and POLLHUP can occur
    * even if only POLLIN was set in events.  (POLLNVAL is a fatal
-   * error and will cause libxl event machinery to fail an assertion.) */
+   * error and will cause libxl event machinery to fail an assertion.)
+   *
+   * It is not permitted to listen for the same or overlapping events
+   * on the same fd using multiple different libxl__ev_fd's.
+   */
 struct libxl__ev_fd {
     /* caller should include this in their own struct */
     /* read-only for caller, who may read only when registered: */
@@ -257,8 +261,8 @@ struct libxl__poller {
     struct pollfd *fd_polls;
     int fd_polls_allocd;
 
-    int fd_rindex_allocd;
-    int *fd_rindex; /* see libxl_osevent_beforepoll */
+    int fd_rindices_allocd;
+    int (*fd_rindices)[3]; /* see libxl_osevent_beforepoll */
 
     int wakeup_pipe[2]; /* 0 means no fd allocated */
 };
-- 
1.7.2.5

[PATCH 22/31] libxl: event API: new facilities for waiting for subprocesses

[Ian Jackson]

The current arrangements in libxl for spawning subprocesses have two
key problems: (i) they make unwarranted (and largely undocumented)
assumptions about the caller's use of subprocesses, (ii) they aren't
integrated into the event system and can't be made asynchronous etc.

So replace them with a new set of facilities.

Primarily, from the point of view of code inside libxl, this is
libxl__ev_child_fork which is both (a) a version of fork() and (b) an
event source which generates a callback when the child dies.

>From the point of view of the application, we fully document our use
of SIGCHLD.  The application can tell us whether it wants to own
SIGCHLD or not; if it does, it has to tell us about deaths of our
children.

Currently there are no callers in libxl which use these facilities.
All code in libxl which forks needs to be converted and libxl_fork
needse to be be abolished.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 tools/libxl/libxl.c          |   17 +++-
 tools/libxl/libxl.h          |    1 +
 tools/libxl/libxl_event.c    |   53 +++++++--
 tools/libxl/libxl_event.h    |  147 +++++++++++++++++++++++-
 tools/libxl/libxl_fork.c     |  255 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   59 +++++++++-
 6 files changed, 512 insertions(+), 20 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 60dbfdc..42ac89f 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -39,7 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     memset(ctx, 0, sizeof(libxl_ctx));
     ctx->lg = lg;
 
-    /* First initialise pointers (cannot fail) */
+    /* First initialise pointers etc. (cannot fail) */
 
     LIBXL_TAILQ_INIT(&ctx->occurred);
 
@@ -58,6 +58,11 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     LIBXL_TAILQ_INIT(&ctx->death_list);
     libxl__ev_xswatch_init(&ctx->death_watch);
 
+    ctx->childproc_hooks = &libxl__childproc_default_hooks;
+    ctx->childproc_user = 0;
+        
+    ctx->sigchld_selfpipe[0] = -1;
+
     /* The mutex is special because we can't idempotently destroy it */
 
     if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
@@ -160,6 +165,16 @@ int libxl_ctx_free(libxl_ctx *ctx)
 
     discard_events(&ctx->occurred);
 
+    /* If we have outstanding children, then the application inherits
+     * them; we wish the application good luck with understanding
+     * this if and when it reaps them. */
+    libxl__sigchld_removehandler(ctx);
+
+    if (ctx->sigchld_selfpipe[0] >= 0) {
+        close(ctx->sigchld_selfpipe[0]);
+        close(ctx->sigchld_selfpipe[1]);
+    }
+
     pthread_mutex_destroy(&ctx->lock);
 
     GC_FREE;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index edbca53..03e71f6 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -380,6 +380,7 @@ enum {
     ERROR_NOT_READY = -11,
     ERROR_OSEVENT_REG_FAIL = -12,
     ERROR_BUFFERFULL = -13,
+    ERROR_UNKNOWN_CHILD = -14,
 };
 
 
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 672e3fe..1b7495a 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -623,6 +623,10 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
                                                                        \
         REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
                                                                        \
+        int selfpipe = libxl__fork_selfpipe_active(CTX);               \
+        if (selfpipe >= 0)                                             \
+            REQUIRE_FD(selfpipe, POLLIN, BODY);                        \
+                                                                       \
     }while(0)
 
 #define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
@@ -762,10 +766,11 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
                                int nfds, const struct pollfd *fds,
                                struct timeval now)
 {
+    /* May make callbacks into the application for child processes.
+     * ctx must be locked exactly once */
     EGC_GC;
     libxl__ev_fd *efd;
 
-
     LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
         if (!efd->events)
             continue;
@@ -776,11 +781,16 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
     }
 
     if (afterpoll_check_fd(poller,fds,nfds, poller->wakeup_pipe[0],POLLIN)) {
-        char buf[256];
-        int r = read(poller->wakeup_pipe[0], buf, sizeof(buf));
-        if (r < 0)
-            if (errno != EINTR && errno != EWOULDBLOCK)
-                LIBXL__EVENT_DISASTER(egc, "read wakeup", errno, 0);
+        int e = libxl__self_pipe_eatall(poller->wakeup_pipe[0]);
+        if (e) LIBXL__EVENT_DISASTER(egc, "read wakeup", e, 0);
+    }
+
+    int selfpipe = libxl__fork_selfpipe_active(CTX);
+    if (selfpipe >= 0 &&
+        afterpoll_check_fd(poller,fds,nfds, selfpipe, POLLIN)) {
+        int e = libxl__self_pipe_eatall(selfpipe);
+        if (e) LIBXL__EVENT_DISASTER(egc, "read sigchld pipe", e, 0);
+        libxl__fork_selfpipe_woken(egc);
     }
 
     for (;;) {
@@ -1078,16 +1088,37 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p)
 
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p)
 {
+    int e = libxl__self_pipe_wakeup(p->wakeup_pipe[1]);
+    if (e) LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", e, 0);
+}
+
+int libxl__self_pipe_wakeup(int fd)
+{
     static const char buf[1] = "";
 
     for (;;) {
-        int r = write(p->wakeup_pipe[1], buf, 1);
-        if (r==1) return;
+        int r = write(fd, buf, 1);
+        if (r==1) return 0;
         assert(r==-1);
         if (errno == EINTR) continue;
-        if (errno == EWOULDBLOCK) return;
-        LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", errno, 0);
-        return;
+        if (errno == EWOULDBLOCK) return 0;
+        assert(errno);
+        return errno;
+    }
+}
+
+int libxl__self_pipe_eatall(int fd)
+{
+    char buf[256];
+    for (;;) {
+        int r = read(fd, buf, sizeof(buf));
+        if (r == sizeof(buf)) continue;
+        if (r >= 0) return 0;
+        assert(r == -1);
+        if (errno == EINTR) continue;
+        if (errno == EWOULDBLOCK) return 0;
+        assert(errno);
+        return errno;
     }
 }
 
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index 2d2196f..713d96d 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -163,11 +163,6 @@ void libxl_event_register_callbacks(libxl_ctx *ctx,
  * After libxl_ctx_free, all corresponding evgen handles become
  * invalid and must no longer be passed to evdisable.
  *
- * Events enabled with evenable prior to a fork and libxl_ctx_postfork
- * are no longer generated after the fork/postfork; however the evgen
- * structures are still valid and must be passed to evdisable if the
- * memory they use should not be leaked.
- *
  * Applications should ensure that they eventually retrieve every
  * event using libxl_event_check or libxl_event_wait, since events
  * which occur but are not retreived by the application will be queued
@@ -372,6 +367,148 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
 void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 
 
+/*======================================================================*/
+
+/*
+ * Subprocess handling.
+ *
+ * Unfortunately the POSIX interface makes this very awkward.
+ *
+ * There are two possible arrangements for collecting statuses from
+ * wait/waitpid.
+ *
+ * For naive programs:
+ *
+ *     libxl will keep a SIGCHLD handler installed whenever it has an
+ *     active (unreaped) child.  It will reap all children with
+ *     wait(); any children it does not recognise will be passed to
+ *     the application via an optional callback (and will result in
+ *     logged warnings if no callback is provided or the callback
+ *     denies responsibility for the child).
+ *
+ *     libxl may have children whenever:
+ *
+ *       - libxl is performing an operation which can be made
+ *         asynchronous; ie one taking a libxl_asyncop_how, even
+ *         if NULL is passed indicating that the operation is
+ *         synchronous; or
+ *
+ *       - events of any kind are being generated, as requested
+ *         by libxl_evenable_....
+ *
+ *     A multithreaded application which is naive in this sense may
+ *     block SIGCHLD on some of its threads, but there must be at
+ *     least one thread that has SIGCHLD unblocked.  libxl will not
+ *     modify the blocking flag for SIGCHLD (except that it may create
+ *     internal service threads with all signals blocked).
+ *
+ *     A naive program must only have at any one time only
+ *     one libxl context which might have children.
+ *
+ * For programs which run their own children alongside libxl's:
+ *
+ *     A program which does this must call libxl_childproc_setmode.
+ *     There are two options:
+ * 
+ *     libxl_sigchld_owner_mainloop:
+ *       The application must install a SIGCHLD handler and reap (at
+ *       least) all of libxl's children and pass their exit status
+ *       to libxl by calling libxl_childproc_exited.
+ *
+ *     libxl_sigchld_owner_libxl_always:
+ *       The application expects libxl to reap all of its children,
+ *       and provides a callback to be notified of their exit
+ *       statues.
+ *
+ * An application which fails to call setmode, or which passes 0 for
+ * hooks, while it uses any libxl operation which might
+ * create or use child processes (see above):
+ *   - Must not have any child processes running.
+ *   - Must not install a SIGCHLD handler.
+ *   - Must not reap any children.
+ */
+
+
+typedef enum {
+    /* libxl owns SIGCHLD whenever it has a child. */
+    libxl_sigchld_owner_libxl,
+
+    /* Application promises to call libxl_childproc_exited but NOT
+     * from within a signal handler.  libxl will not itself arrange to
+     * (un)block or catch SIGCHLD. */
+    libxl_sigchld_owner_mainloop,
+
+    /* libxl owns SIGCHLD all the time, and the application is
+     * relying on libxl's event loop for reaping its own children. */
+    libxl_sigchld_owner_libxl_always,
+} libxl_sigchld_owner;
+
+typedef struct {
+    libxl_sigchld_owner chldowner;
+
+    /* All of these are optional: */
+
+    /* Called by libxl instead of fork.  Should behave exactly like
+     * fork, including setting errno etc.  May NOT reenter into libxl.
+     * Application may use this to discover pids of libxl's children,
+     * for example.
+     */
+    pid_t (*fork_replacement)(void *user);
+
+    /* With libxl_sigchld_owner_libxl, called by libxl when it has
+     * reaped a pid.  (Not permitted with _owner_mainloop.)
+     *
+     * Should return 0 if the child was recognised by the application
+     * (or if the application does not keep those kind of records),
+     * ERROR_UNKNOWN_CHILD if the application knows that the child is not
+     * the application's; if it returns another error code it is a
+     * disaster as described for libxl_event_register_callbacks.
+     * (libxl will report unexpected children to its error log.)
+     *
+     * If not supplied, the application is assumed not to start
+     * any children of its own.
+     *
+     * This function is NOT called from within the signal handler.
+     * Rather it will be called from inside a libxl's event handling
+     * code and thus only when libxl is running, for example from
+     * within libxl_event_wait.  (libxl uses the self-pipe trick
+     * to implement this.)
+     *
+     * childproc_exited_callback may call back into libxl, but it
+     * is best to avoid making long-running libxl calls as that might
+     * stall the calling event loop while the nested operation
+     * completes.
+     */
+    int (*reaped_callback)(pid_t, int status, void *user);
+} libxl_childproc_hooks;
+
+/* hooks may be 0 in which is equivalent to &{ libxl_sigchld_owner_libxl, 0, 0 }
+ *
+ * May not be called when libxl might have any child processes, or the
+ * behaviour is undefined.  So it is best to call this at
+ * initialisation.
+ */
+void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
+                             void *user);
+
+/*
+ * This function is for an application which owns SIGCHLD and which
+ * therefore reaps all of the process's children.
+ *
+ * May be called only by an application which has called setmode with
+ * chldowner == libxl_sigchld_owner_mainloop.  If pid was a process started
+ * by this instance of libxl, returns 0 after doing whatever
+ * processing is appropriate.  Otherwise silently returns
+ * ERROR_UNKNOWN_CHILD.  No other error returns are possible.
+ *
+ * May NOT be called from within a signal handler which might
+ * interrupt any libxl operation.  The application will almost
+ * certainly need to use the self-pipe trick (or a working pselect or
+ * ppoll) to implement this.
+ */
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status);
+
+
 /*
  * An application which initialises a libxl_ctx in a parent process
  * and then forks a child which does not quickly exec, must
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
index 4751ef4..aac9598 100644
--- a/tools/libxl/libxl_fork.c
+++ b/tools/libxl/libxl_fork.c
@@ -46,6 +46,12 @@ static int atfork_registered;
 static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
     LIBXL_LIST_HEAD_INITIALIZER(carefds);
 
+/* non-null iff installed, protected by no_forking */
+static libxl_ctx *sigchld_owner;
+static struct sigaction sigchld_saved_action;
+
+static void sigchld_removehandler_core(void);
+
 static void atfork_lock(void)
 {
     int r = pthread_mutex_lock(&no_forking);
@@ -104,6 +110,7 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
     int r;
 
     atfork_lock();
+
     LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
         if (cf->fd >= 0) {
             r = close(cf->fd);
@@ -115,6 +122,10 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
         free(cf);
     }
     LIBXL_LIST_INIT(&carefds);
+
+    if (sigchld_owner)
+        sigchld_removehandler_core();
+
     atfork_unlock();
 }
 
@@ -138,6 +149,250 @@ int libxl__carefd_fd(const libxl__carefd *cf)
 }
 
 /*
+ * Actual child process handling
+ */
+
+static void sigchld_handler(int signo)
+{
+    int e = libxl__self_pipe_wakeup(sigchld_owner->sigchld_selfpipe[1]);
+    assert(!e); /* errors are probably EBADF, very bad */
+}
+
+static void sigchld_removehandler_core(void)
+{
+    struct sigaction was;
+    int r;
+    
+    r = sigaction(SIGCHLD, &sigchld_saved_action, &was);
+    assert(!r);
+    assert(!(was.sa_flags & SA_SIGINFO));
+    assert(was.sa_handler == sigchld_handler);
+    sigchld_owner = 0;
+}
+
+void libxl__sigchld_removehandler(libxl_ctx *ctx) /* non-reentrant */
+{
+    atfork_lock();
+    if (sigchld_owner == ctx)
+        sigchld_removehandler_core();
+    atfork_unlock();
+}
+
+int libxl__sigchld_installhandler(libxl_ctx *ctx) /* non-reentrant */
+{
+    int r, rc;
+
+    if (ctx->sigchld_selfpipe[0] < 0) {
+        r = pipe(ctx->sigchld_selfpipe);
+        if (r) {
+            ctx->sigchld_selfpipe[0] = -1;
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+                             "failed to create sigchld pipe");
+            rc = ERROR_FAIL;
+            goto out;
+        }
+    }
+
+    atfork_lock();
+    if (sigchld_owner != ctx) {
+        struct sigaction ours;
+
+        assert(!sigchld_owner);
+        sigchld_owner = ctx;
+
+        memset(&ours,0,sizeof(ours));
+        ours.sa_handler = sigchld_handler;
+        sigemptyset(&ours.sa_mask);
+        ours.sa_flags = SA_NOCLDSTOP | SA_RESTART;
+        r = sigaction(SIGCHLD, &ours, &sigchld_saved_action);
+        assert(!r);
+
+        assert(((void)"application must negotiate with libxl about SIGCHLD",
+                !(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
+                (sigchld_saved_action.sa_handler == SIG_DFL ||
+                 sigchld_saved_action.sa_handler == SIG_IGN)));
+    }
+    atfork_unlock();
+
+    rc = 0;
+ out:
+    return rc;
+}
+
+static int chldmode_ours(libxl_ctx *ctx)
+{
+    return ctx->childproc_hooks->chldowner == libxl_sigchld_owner_libxl;
+}
+
+int libxl__fork_selfpipe_active(libxl_ctx *ctx)
+{
+    /* Returns the fd to read, or -1 */
+    if (!chldmode_ours(ctx))
+        return -1;
+
+    if (LIBXL_LIST_EMPTY(&ctx->children))
+        return -1;
+
+    return ctx->sigchld_selfpipe[0];
+}
+
+static void perhaps_removehandler(libxl_ctx *ctx)
+{
+    if (LIBXL_LIST_EMPTY(&ctx->children) &&
+        ctx->childproc_hooks->chldowner != libxl_sigchld_owner_libxl_always)
+        libxl__sigchld_removehandler(ctx);
+}
+
+static int childproc_reaped(libxl__egc *egc, pid_t pid, int status)
+{
+    EGC_GC;
+    libxl__ev_child *ch;
+
+    LIBXL_LIST_FOREACH(ch, &CTX->children, entry)
+        if (ch->pid == pid)
+            goto found;
+
+    /* not found */
+    return ERROR_UNKNOWN_CHILD;
+
+ found:
+    LIBXL_LIST_REMOVE(ch, entry);
+    ch->pid = -1;
+    ch->callback(egc, ch, pid, status);
+
+    perhaps_removehandler(CTX);
+
+    return 0;
+}
+
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t pid, int status)
+{
+    EGC_INIT(ctx);
+    CTX_LOCK;
+    int rc = childproc_reaped(egc, pid, status);
+    CTX_UNLOCK;
+    EGC_FREE;
+    return rc;
+}
+
+void libxl__fork_selfpipe_woken(libxl__egc *egc)
+{
+    /* May make callbacks into the application for child processes.
+     * ctx must be locked EXACTLY ONCE */
+    EGC_GC;
+
+    while (chldmode_ours(CTX) /* in case the app changes the mode */) {
+        int status;
+        pid_t pid = waitpid(-1, &status, WNOHANG);
+
+        if (pid == 0) return;
+
+        if (pid == -1) {
+            if (errno == ECHILD) return;
+            if (errno == EINTR) continue;
+            LIBXL__EVENT_DISASTER(egc, "waitpid() failed", errno, 0);
+            return;
+        }
+
+        int rc = childproc_reaped(egc, pid, status);
+
+        if (rc) {
+            if (CTX->childproc_hooks->reaped_callback) {
+                CTX_UNLOCK;
+                rc = CTX->childproc_hooks->reaped_callback
+                        (pid, status, CTX->childproc_user);
+                CTX_LOCK;
+                if (rc != 0 && rc != ERROR_UNKNOWN_CHILD) {
+                    char disasterbuf[200];
+                    snprintf(disasterbuf, sizeof(disasterbuf), " reported by"
+                             " libxl_childproc_hooks->reaped_callback"
+                             " (for pid=%lu, status=%d; error code %d)",
+                             (unsigned long)pid, status, rc);
+                    LIBXL__EVENT_DISASTER(egc, disasterbuf, 0, 0);
+                    return;
+                }
+            } else {
+                rc = ERROR_UNKNOWN_CHILD;
+            }
+            if (rc)
+                libxl_report_child_exitstatus(CTX, XTL_WARN,
+                                "unknown child", (long)pid, status);
+        }
+    }
+}
+
+pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *ch,
+                           libxl__ev_child_callback *death)
+{
+    CTX_LOCK;
+    int rc;
+
+    if (chldmode_ours(CTX)) {
+        rc = libxl__sigchld_installhandler(CTX);
+        if (rc) goto out;
+    }
+
+    pid_t pid =
+        CTX->childproc_hooks->fork_replacement
+        ? CTX->childproc_hooks->fork_replacement(CTX->childproc_user)
+        : fork();
+    if (pid == -1) {
+        LOGE(ERROR, "fork failed");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    if (!pid) {
+        /* woohoo! */
+        return 0; /* Yes, CTX is left locked in the child. */
+    }
+
+    ch->pid = pid;
+    ch->callback = death;
+    LIBXL_LIST_INSERT_HEAD(&CTX->children, ch, entry);
+    rc = pid;
+
+ out:
+    perhaps_removehandler(CTX);
+    CTX_UNLOCK;
+    return rc;
+}
+
+void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
+                             void *user)
+{
+    GC_INIT(ctx);
+    CTX_LOCK;
+
+    assert(LIBXL_LIST_EMPTY(&CTX->children));
+
+    if (!hooks)
+        hooks = &libxl__childproc_default_hooks;
+
+    ctx->childproc_hooks = hooks;
+    ctx->childproc_user = user;
+
+    switch (ctx->childproc_hooks->chldowner) {
+    case libxl_sigchld_owner_mainloop:
+    case libxl_sigchld_owner_libxl:
+        libxl__sigchld_removehandler(ctx);
+        break;
+    case libxl_sigchld_owner_libxl_always:
+        libxl__sigchld_installhandler(ctx);
+        break;
+    default:
+        abort();
+    }
+
+    CTX_UNLOCK;
+    GC_FREE;
+}
+
+const libxl_childproc_hooks libxl__childproc_default_hooks = {
+    libxl_sigchld_owner_libxl, 0, 0
+};
+
+/*
  * Local variables:
  * mode: C
  * c-basic-offset: 4
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 1a2139c..a60171c 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -196,6 +196,19 @@ typedef struct libxl__ev_watch_slot {
 libxl__ev_xswatch *libxl__watch_slot_contents(libxl__gc *gc, int slotnum);
 
 
+typedef struct libxl__ev_child libxl__ev_child;
+typedef void libxl__ev_child_callback(libxl__egc *egc, libxl__ev_child*,
+                                      pid_t pid, int status);
+struct libxl__ev_child {
+    /* caller should include this in their own struct */
+    /* read-only for caller: */
+    pid_t pid; /* -1 means unused ("unregistered", ie Idle) */
+    libxl__ev_child_callback *callback;
+    /* remainder is private for libxl__ev_... */
+    LIBXL_LIST_ENTRY(struct libxl__ev_child) entry;
+};
+
+
 /*
  * evgen structures, which are the state we use for generating
  * events for the caller.
@@ -315,10 +328,14 @@ struct libxl__ctx {
     
     LIBXL_LIST_HEAD(, libxl_evgen_disk_eject) disk_eject_evgens;
 
-    /* for callers who reap children willy-nilly; caller must only
-     * set this after libxl_init and before any other call - or
-     * may leave them untouched */
+    const libxl_childproc_hooks *childproc_hooks;
+    void *childproc_user;
+    int sigchld_selfpipe[2]; /* [0]==-1 means handler not installed */
+    LIBXL_LIST_HEAD(, libxl__ev_child) children;
+
+    /* This is obsolete and must be removed: */
     int (*waitpid_instead)(pid_t pid, int *status, int flags);
+
     libxl_version_info version_info;
 };
 
@@ -566,6 +583,33 @@ static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw)
 
 
 /*
+ * For making subprocesses.  This is the only permitted mechanism for
+ * code in libxl to do so.
+ *
+ * In the parent, returns the pid, filling in childw_out.
+ * In the child, returns 0.
+ * If it fails, returns a libxl error (all of which are -ve).
+ *
+ * The child should go on to exec (or exit) soon, and should not make
+ * any further libxl event calls in the meantime.
+ *
+ * The parent may signal the child but it must not reap it.  That will
+ * be done by the event machinery.  death may be NULL in which case
+ * the child is still reaped but its death is ignored.
+ *
+ * It is not possible to "deregister" the child death event source.
+ * It will generate exactly one event callback; until then the childw
+ * is Active and may not be reused.
+ */
+_hidden pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *childw_out,
+                                 libxl__ev_child_callback *death);
+static inline void libxl__ev_child_init(libxl__ev_child *childw_out)
+                { childw_out->pid = -1; }
+static inline int libxl__ev_child_inuse(libxl__ev_child *childw_out)
+                { return childw_out->pid >= 0; }
+
+
+/*
  * Other event-handling support provided by the libxl event core to
  * the rest of libxl.
  */
@@ -619,6 +663,15 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
  * ctx must be locked. */
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
 
+/* Internal to fork and child reaping machinery */
+extern const libxl_childproc_hooks libxl__childproc_default_hooks;
+int libxl__sigchld_installhandler(libxl_ctx *ctx); /* non-reentrant;logs errs */
+void libxl__sigchld_removehandler(libxl_ctx *ctx); /* non-reentrant */
+int libxl__fork_selfpipe_active(libxl_ctx *ctx); /* returns read fd or -1 */
+void libxl__fork_selfpipe_woken(libxl__egc *egc);
+int libxl__self_pipe_wakeup(int fd); /* returns 0 or -1 setting errno */
+int libxl__self_pipe_eatall(int fd); /* returns 0 or -1 setting errno */
+
 
 int libxl__atfork_init(libxl_ctx *ctx);
 
-- 
1.7.2.5

[PATCH 23/31] autoconf: New test for openpty et al.

[Ian Jackson]

We may need to #include <libutil.h>, and/or link with -lutil, to use
openpty, login_tty, and the like.  Provide INCLUDE_LIBUTIL_H
(preprocessor constant, not always defined) and PTYFUNCS_LIBS
(makefile variable).

We link libxl against PTYFUNCS_LIBS (which comes from autoconf) rather
than UTIL_LIBS, and #include <libutil.h> where appropriate.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 config/Tools.mk.in             |    2 +
 tools/configure                |   51 ++++++++++++++++++++++++++++++++++++++++
 tools/configure.ac             |    2 +
 tools/libxl/Makefile           |    2 +-
 tools/libxl/libxl_bootloader.c |    4 +++
 tools/m4/ptyfuncs.m4           |   24 ++++++++++++++++++
 6 files changed, 84 insertions(+), 1 deletions(-)
 create mode 100644 tools/m4/ptyfuncs.m4

diff --git a/config/Tools.mk.in b/config/Tools.mk.in
index 912d021..eb879dd 100644
--- a/config/Tools.mk.in
+++ b/config/Tools.mk.in
@@ -30,6 +30,8 @@ PTHREAD_CFLAGS      := @PTHREAD_CFLAGS@
 PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
 PTHREAD_LIBS        := @PTHREAD_LIBS@
 
+PTYFUNCS_LIBS       := @PTYFUNCS_LIBS@
+
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
 # may block it). We make it the default, but if your GIT repository downloads
diff --git a/tools/configure b/tools/configure
index 86618f5..eeaf901 100755
--- a/tools/configure
+++ b/tools/configure
@@ -602,6 +602,7 @@ POW_LIB
 LIBOBJS
 ALLOCA
 libiconv
+PTYFUNCS_LIBS
 PTHREAD_LIBS
 PTHREAD_LDFLAGS
 PTHREAD_CFLAGS
@@ -3947,6 +3948,8 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
 
 
 
+
+
 # Enable/disable options
 
 # Check whether --enable-githttp was given.
@@ -7315,6 +7318,54 @@ $as_echo "$ax_cv_pthread_flags" >&6; }
 
 
 
+
+    ac_fn_c_check_header_mongrel "$LINENO" "libutil.h" "ac_cv_header_libutil_h" "$ac_includes_default"
+if test "x$ac_cv_header_libutil_h" = x""yes; then :
+
+
+$as_echo "#define INCLUDE_LIBUTIL_H <libutil.h>" >>confdefs.h
+
+
+fi
+
+
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty et al" >&5
+$as_echo_n "checking for openpty et al... " >&6; }
+if test "${ax_cv_ptyfuncs_libs+set}" = set; then :
+  $as_echo_n "(cached) " >&6
+else
+
+        ax_cv_ptyfuncs_libs=-lutil
+
+    saved_LIBS="$LIBS"
+
+        LIBS="$LIBS $ax_cv_ptyfuncs_libs"
+        cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+#ifdef INCLUDE_LIBUTIL_H
+#include INCLUDE_LIBUTIL_H
+#endif
+int main(void) {
+  openpty(0,0,0,0,0);
+  login_tty(0);
+}
+
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+
+    LIBS="$saved_LIBS"
+
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ax_cv_ptyfuncs_libs" >&5
+$as_echo "$ax_cv_ptyfuncs_libs" >&6; }
+    PTYFUNCS_LIBS="$ax_cv_ptyfuncs_libs"
+
+
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_gettime in -lrt" >&5
 $as_echo_n "checking for clock_gettime in -lrt... " >&6; }
 if test "${ac_cv_lib_rt_clock_gettime+set}" = set; then :
diff --git a/tools/configure.ac b/tools/configure.ac
index 250dffd..b4f72e8 100644
--- a/tools/configure.ac
+++ b/tools/configure.ac
@@ -35,6 +35,7 @@ m4_include([m4/uuid.m4])
 m4_include([m4/pkg.m4])
 m4_include([m4/curses.m4])
 m4_include([m4/pthread.m4])
+m4_include([m4/ptyfuncs.m4])
 
 # Enable/disable options
 AX_ARG_DEFAULT_DISABLE([githttp], [Download GIT repositories via HTTP])
@@ -132,6 +133,7 @@ AC_SUBST(libext2fs)
 AC_CHECK_LIB([gcrypt], [gcry_md_hash_buffer], [libgcrypt="y"], [libgcrypt="n"])
 AC_SUBST(libgcrypt)
 AX_CHECK_PTHREAD
+AX_CHECK_PTYFUNCS
 AC_CHECK_LIB([rt], [clock_gettime])
 AC_CHECK_LIB([yajl], [yajl_alloc], [],
     [AC_MSG_ERROR([Could not find yajl])])
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index e5ea867..5ba144f 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -20,7 +20,7 @@ LIBUUID_LIBS += -luuid
 endif
 
 LIBXL_LIBS =
-LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(UTIL_LIBS) $(LIBUUID_LIBS)
+LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS)
 
 CFLAGS += $(PTHREAD_CFLAGS)
 LDFLAGS += $(PTHREAD_LDFLAGS)
diff --git a/tools/libxl/libxl_bootloader.c b/tools/libxl/libxl_bootloader.c
index 2774062..b50944a 100644
--- a/tools/libxl/libxl_bootloader.c
+++ b/tools/libxl/libxl_bootloader.c
@@ -16,6 +16,10 @@
 
 #include <termios.h>
 
+#ifdef INCLUDE_LIBUTIL_H
+#include INCLUDE_LIBUTIL_H
+#endif
+
 #include "libxl_internal.h"
 
 #define XENCONSOLED_BUF_SIZE 16
diff --git a/tools/m4/ptyfuncs.m4 b/tools/m4/ptyfuncs.m4
new file mode 100644
index 0000000..2846a6d
--- /dev/null
+++ b/tools/m4/ptyfuncs.m4
@@ -0,0 +1,24 @@
+AC_DEFUN([AX_CHECK_PTYFUNCS], [
+    AC_CHECK_HEADER([libutil.h],[
+      AC_DEFINE([INCLUDE_LIBUTIL_H],[<libutil.h>],[libutil header file name])
+    ])
+    AC_CACHE_CHECK([for openpty et al], [ax_cv_ptyfuncs_libs], [
+        ax_cv_ptyfuncs_libs=-lutil
+        AX_SAVEVAR_SAVE(LIBS)
+        LIBS="$LIBS $ax_cv_ptyfuncs_libs"
+        AC_LINK_IFELSE([
+#ifdef INCLUDE_LIBUTIL_H
+#include INCLUDE_LIBUTIL_H
+#endif
+int main(void) {
+  openpty(0,0,0,0,0);
+  login_tty(0);
+}
+])
+        AX_SAVEVAR_RESTORE(LIBS)],
+    [],[
+        AC_MSG_FAILURE([Unable to find -lutil for openpty and login_tty])
+    ])
+    PTYFUNCS_LIBS="$ax_cv_ptyfuncs_libs"
+    AC_SUBST(PTYFUNCS_LIBS)
+])
-- 
1.7.2.5

[PATCH 24/31] libxl: provide libxl__remove_file et al.

[Ian Jackson]

These utility functions cope with EINTR AND ENOENT, do error logging,
and we provide a recursive version to delete whole directory trees.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_internal.h |    7 ++++
 tools/libxl/libxl_utils.c    |   79 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index a60171c..8e47213 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -442,6 +442,13 @@ _hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n);
  * string. (similar to a gc'd dirname(3)). */
 _hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s);
 
+/* Each of these logs errors and returns a libxl error code.
+ * They do not mind if path is already removed.
+ * For _file, path must not be a directory; for _directory it must be. */
+_hidden int libxl__remove_file(libxl__gc *gc, const char *path);
+_hidden int libxl__remove_directory(libxl__gc *gc, const char *path);
+_hidden int libxl__remove_file_or_directory(libxl__gc *gc, const char *path);
+
 _hidden char **libxl__xs_kvs_of_flexarray(libxl__gc *gc, flexarray_t *array, int length);
 
 _hidden int libxl__xs_writev(libxl__gc *gc, xs_transaction_t t,
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index 0cbd85e..c9157a6 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -364,6 +364,85 @@ int libxl_read_file_contents(libxl_ctx *ctx, const char *filename,
 READ_WRITE_EXACTLY(read, 1, /* */)
 READ_WRITE_EXACTLY(write, 0, const)
 
+int libxl__remove_file(libxl__gc *gc, const char *path)
+{
+    for (;;) {
+        int r = unlink(path);
+        if (!r) return 0;
+        if (errno == ENOENT) return 0;
+        if (errno == EINTR) continue;
+        LOGE(ERROR, "failed to remove file %s", path);
+        return ERROR_FAIL;
+     }
+}
+
+int libxl__remove_file_or_directory(libxl__gc *gc, const char *path)
+{
+    for (;;) {
+        int r = rmdir(path);
+        if (!r) return 0;
+        if (errno == ENOENT) return 0;
+        if (errno == ENOTEMPTY) return libxl__remove_directory(gc, path);
+        if (errno == ENOTDIR) return libxl__remove_file(gc, path);
+        if (errno == EINTR) continue;
+        LOGE(ERROR, "failed to remove %s", path);
+        return ERROR_FAIL;
+     }
+}
+
+int libxl__remove_directory(libxl__gc *gc, const char *dirpath)
+{
+    int rc = 0;
+    DIR *d = 0;
+
+    d = opendir(dirpath);
+    if (!d) {
+        if (errno == ENOENT)
+            goto out;
+
+        LOGE(ERROR, "failed to opendir %s for removal", dirpath);
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    size_t need = offsetof(struct dirent, d_name) +
+        pathconf(dirpath, _PC_NAME_MAX) + 1;
+    struct dirent *de_buf = libxl__zalloc(gc, need);
+    struct dirent *de;
+
+    for (;;) {
+        int r = readdir_r(d, de_buf, &de);
+        if (r) {
+            LOGE(ERROR, "failed to readdir %s for removal", dirpath);
+            rc = ERROR_FAIL;
+            break;
+        }
+        if (!de)
+            break;
+
+        if (!strcmp(de->d_name, ".") ||
+            !strcmp(de->d_name, ".."))
+            continue;
+
+        const char *subpath = GCSPRINTF("%s/%s", dirpath, de->d_name);
+        if (libxl__remove_file_or_directory(gc, subpath))
+            rc = ERROR_FAIL;
+    }
+
+    for (;;) {
+        int r = rmdir(dirpath);
+        if (!r) break;
+        if (errno == ENOENT) return 0;
+        if (errno == EINTR) continue;
+        LOGE(ERROR, "failed to remove emptied directory %s", dirpath);
+        rc = ERROR_FAIL;
+    }
+
+ out:
+    if (d) closedir(d);
+
+    return rc;
+}
 
 pid_t libxl_fork(libxl_ctx *ctx)
 {
-- 
1.7.2.5

[PATCH 25/31] libxl: Introduce libxl__sendmsg_fds and libxl__recvmsg_fds

[Ian Jackson]

We will want to reuse the fd-sending code, so break it out into its
own function, and provide the corresponding sending function.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl_internal.h |   11 ++++
 tools/libxl/libxl_qmp.c      |   31 ++-----------
 tools/libxl/libxl_utils.c    |  104 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 119 insertions(+), 27 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8e47213..a3955a4 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1125,6 +1125,17 @@ _hidden void libxl__qmp_cleanup(libxl__gc *gc, uint32_t domid);
 _hidden int libxl__qmp_initializations(libxl__gc *gc, uint32_t domid,
                                        const libxl_domain_config *guest_config);
 
+/* on failure, logs */
+int libxl__sendmsg_fds(libxl__gc *gc, int carrier,
+                       const void *data, size_t datalen,
+                       int nfds, const int fds[], const char *what);
+
+/* Insists on receiving exactly nfds and datalen.  On failure, logs
+ * and leaves *fds untouched. */
+int libxl__recvmsg_fds(libxl__gc *gc, int carrier,
+                       void *databuf, size_t datalen,
+                       int nfds, int fds[], const char *what);
+
 /* from libxl_json */
 #include <yajl/yajl_gen.h>
 
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index f5a3edc..83c22b3 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -544,38 +544,15 @@ static int qmp_send_fd(libxl__gc *gc, libxl__qmp_handler *qmp,
                        qmp_request_context *context,
                        int fd)
 {
-    struct msghdr msg = { 0 };
-    struct cmsghdr *cmsg;
-    char control[CMSG_SPACE(sizeof (fd))];
-    struct iovec iov;
     char *buf = NULL;
+    int rc;
 
     buf = qmp_send_prepare(gc, qmp, "getfd", args, callback, opaque, context);
 
-    /* Response data */
-    iov.iov_base = buf;
-    iov.iov_len  = strlen(buf);
-
-    /* compose the message */
-    msg.msg_iov = &iov;
-    msg.msg_iovlen = 1;
-    msg.msg_control = control;
-    msg.msg_controllen = sizeof (control);
-
-    /* attach open fd */
-    cmsg = CMSG_FIRSTHDR(&msg);
-    cmsg->cmsg_level = SOL_SOCKET;
-    cmsg->cmsg_type = SCM_RIGHTS;
-    cmsg->cmsg_len = CMSG_LEN(sizeof (fd));
-    *(int *)CMSG_DATA(cmsg) = fd;
-
-    msg.msg_controllen = cmsg->cmsg_len;
+    rc = libxl__sendmsg_fds(gc, qmp->qmp_fd, buf, strlen(buf), 1, &fd,
+                            "QMP message to QEMU");
+    if (rc) return rc;
 
-    if (sendmsg(qmp->qmp_fd, &msg, 0) < 0) {
-        LIBXL__LOG_ERRNO(qmp->ctx, LIBXL__LOG_ERROR,
-                         "Failed to send a QMP message to QEMU.");
-        return ERROR_FAIL;
-    }
     if (libxl_write_exactly(qmp->ctx, qmp->qmp_fd, "\r\n", 2,
                             "CRLF", "QMP socket")) {
         return ERROR_FAIL;
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index c9157a6..dc25ab0 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -579,6 +579,110 @@ void libxl_cputopology_list_free(libxl_cputopology *list, int nr)
     free(list);
 }
 
+int libxl__sendmsg_fds(libxl__gc *gc, int carrier,
+                       const void *data, size_t datalen,
+                       int nfds, const int fds[], const char *what) {
+    struct msghdr msg = { 0 };
+    struct cmsghdr *cmsg;
+    size_t spaceneeded = nfds * sizeof(fds[0]);
+    char control[CMSG_SPACE(spaceneeded)];
+    struct iovec iov;
+    int r;
+
+    iov.iov_base = (void*)data;
+    iov.iov_len  = datalen;
+
+    /* compose the message */
+    msg.msg_iov = &iov;
+    msg.msg_iovlen = 1;
+    msg.msg_control = control;
+    msg.msg_controllen = sizeof(control);
+
+    /* attach open fd */
+    cmsg = CMSG_FIRSTHDR(&msg);
+    cmsg->cmsg_level = SOL_SOCKET;
+    cmsg->cmsg_type = SCM_RIGHTS;
+    cmsg->cmsg_len = CMSG_LEN(spaceneeded);
+    memcpy(CMSG_DATA(cmsg), fds, spaceneeded);
+
+    msg.msg_controllen = cmsg->cmsg_len;
+
+    r = sendmsg(carrier, &msg, 0);
+    if (r < 0) {
+        LOGE(ERROR, "failed to send fd-carrying message (%s)", what);
+        return ERROR_FAIL;
+    }
+
+    return 0;
+}
+
+int libxl__recvmsg_fds(libxl__gc *gc, int carrier,
+                       void *databuf, size_t datalen,
+                       int nfds, int fds[], const char *what)
+{
+    struct msghdr msg = { 0 };
+    struct cmsghdr *cmsg;
+    size_t spaceneeded = nfds * sizeof(fds[0]);
+    char control[CMSG_SPACE(spaceneeded)];
+    struct iovec iov;
+    int r;
+
+    iov.iov_base = databuf;
+    iov.iov_len  = datalen;
+
+    msg.msg_iov = &iov;
+    msg.msg_iovlen = 1;
+    msg.msg_control = control;
+    msg.msg_controllen = sizeof(control);
+
+    for (;;) {
+        r = recvmsg(carrier, &msg, 0);
+        if (r < 0) {
+            if (errno == EINTR) continue;
+            if (errno == EWOULDBLOCK) return -1;
+            LOGE(ERROR,"recvmsg failed (%s)", what);
+            return ERROR_FAIL;
+        }
+        if (r == 0) {
+            LOG(ERROR,"recvmsg got EOF (%s)", what);
+            return ERROR_FAIL;
+        }
+        cmsg = CMSG_FIRSTHDR(&msg);
+        if (cmsg->cmsg_len <= CMSG_LEN(0)) {
+            LOG(ERROR,"recvmsg got no control msg"
+                " when expecting fds (%s)", what);
+            return ERROR_FAIL;
+        }
+        if (cmsg->cmsg_level != SOL_SOCKET || cmsg->cmsg_type != SCM_RIGHTS) {
+            LOG(ERROR, "recvmsg got unexpected"
+                " cmsg_level %d (!=%d) or _type %d (!=%d) (%s)",
+                cmsg->cmsg_level, SOL_SOCKET,
+                cmsg->cmsg_type, SCM_RIGHTS,
+                what);
+            return ERROR_FAIL;
+        }
+        if (cmsg->cmsg_len != CMSG_LEN(spaceneeded) ||
+            msg.msg_controllen != cmsg->cmsg_len) {
+            LOG(ERROR, "recvmsg got unexpected"
+                " number of fds or extra control data"
+                " (%ld bytes' worth, expected %ld) (%s)",
+                (long)CMSG_LEN(spaceneeded), (long)cmsg->cmsg_len,
+                what);
+            int i, fd;
+            unsigned char *p;
+            for (i=0, p=CMSG_DATA(cmsg);
+                 CMSG_SPACE(i * sizeof(fds[0]));
+                 i++, i+=sizeof(fd)) {
+                memcpy(&fd, p, sizeof(fd));
+                close(fd);
+            }
+            return ERROR_FAIL;
+        }
+        memcpy(fds, CMSG_DATA(cmsg), spaceneeded);
+        return 0;
+    }
+}         
+
 void libxl_dominfo_list_free(libxl_dominfo *list, int nr)
 {
     int i;
-- 
1.7.2.5

[PATCH 26/31] libxl: Clean up setdefault in do_domain_create

[Ian Jackson]

do_domain_create called libxl__domain_create_info_setdefault twice.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Ian Campbell <Ian.Campbell@citrix.com>
---
 tools/libxl/libxl_create.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index e63c7bd..3675afe 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -552,9 +552,6 @@ static int do_domain_create(libxl__gc *gc, libxl_domain_config *d_config,
     ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
     if (ret) goto error_out;
 
-    ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
-    if (ret) goto error_out;
-
     ret = libxl__domain_make(gc, &d_config->c_info, &domid);
     if (ret) {
         LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "cannot make domain: %d", ret);
-- 
1.7.2.5

[PATCH 27/31] libxl: provide libxl__datacopier_*

[Ian Jackson]

General facility for ao operations to shovel data between fds.

This will be used by the bootloader machinery.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/Makefile         |    3 +-
 tools/libxl/libxl_aoutils.c  |  189 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   40 +++++++++
 3 files changed, 231 insertions(+), 1 deletions(-)
 create mode 100644 tools/libxl/libxl_aoutils.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 5ba144f..6e253b1 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -52,7 +52,8 @@ LIBXL_LIBS += -lyajl
 
 LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
 			libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
-			libxl_internal.o libxl_utils.o libxl_uuid.o libxl_json.o \
+			libxl_internal.o libxl_utils.o libxl_uuid.o \
+			libxl_json.o libxl_aoutils.o \
 			libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
 LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
 
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
new file mode 100644
index 0000000..c1229e4
--- /dev/null
+++ b/tools/libxl/libxl_aoutils.c
@@ -0,0 +1,189 @@
+/*
+ * Copyright (C) 2010      Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*----- data copier -----*/
+
+void libxl__datacopier_init(libxl__datacopier_state *dc)
+{
+    libxl__ev_fd_init(&dc->toread);
+    libxl__ev_fd_init(&dc->towrite);
+    LIBXL_TAILQ_INIT(&dc->bufs);
+}
+
+void libxl__datacopier_kill(libxl__datacopier_state *dc)
+{
+    STATE_AO_GC(dc);
+    libxl__datacopier_buf *buf, *tbuf;
+
+    libxl__ev_fd_deregister(gc, &dc->toread);
+    libxl__ev_fd_deregister(gc, &dc->towrite);
+    LIBXL_TAILQ_FOREACH_SAFE(buf, &dc->bufs, entry, tbuf)
+        free(buf);
+    LIBXL_TAILQ_INIT(&dc->bufs);
+}
+
+static void datacopier_callback(libxl__egc *egc, libxl__datacopier_state *dc,
+                                int onwrite, int errnoval)
+{
+    libxl__datacopier_kill(dc);
+    dc->callback(egc, dc, onwrite, errnoval);
+}
+
+static void datacopier_writable(libxl__egc *egc, libxl__ev_fd *ev,
+                                int fd, short events, short revents);
+
+static void datacopier_check_state(libxl__egc *egc, libxl__datacopier_state *dc)
+{
+    STATE_AO_GC(dc);
+    int rc;
+    
+    if (dc->used) {
+        if (!libxl__ev_fd_isregistered(&dc->towrite)) {
+            rc = libxl__ev_fd_register(gc, &dc->towrite, datacopier_writable,
+                                       dc->writefd, POLLOUT);
+            if (rc) {
+                LOG(ERROR, "unable to establish write event on %s"
+                    " during copy of %s", dc->writewhat, dc->copywhat);
+                datacopier_callback(egc, dc, -1, 0);
+                return;
+            }
+        }
+    } else if (!libxl__ev_fd_isregistered(&dc->toread)) {
+        /* we have had eof */
+        libxl__datacopier_kill(dc);
+        dc->callback(egc, dc, 0, 0);
+        return;
+    } else {
+        /* nothing buffered, but still reading */
+        libxl__ev_fd_deregister(gc, &dc->towrite);
+    }
+}
+
+static void datacopier_readable(libxl__egc *egc, libxl__ev_fd *ev,
+                                int fd, short events, short revents) {
+    libxl__datacopier_state *dc = CONTAINER_OF(ev, *dc, toread);
+    STATE_AO_GC(dc);
+
+    if (revents & ~POLLIN) {
+        LOG(ERROR, "unexpected poll event 0x%x (should be POLLIN)"
+            " on %s during copy of %s", revents, dc->readwhat, dc->copywhat);
+        datacopier_callback(egc, dc, -1, 0);
+        return;
+    }
+    assert(revents & POLLIN);
+    for (;;) {
+        while (dc->used >= dc->maxsz) {
+            libxl__datacopier_buf *rm = LIBXL_TAILQ_FIRST(&dc->bufs);
+            dc->used -= rm->used;
+            assert(dc->used >= 0);
+            LIBXL_TAILQ_REMOVE(&dc->bufs, rm, entry);
+            free(rm);
+        }
+
+        libxl__datacopier_buf *buf =
+            LIBXL_TAILQ_LAST(&dc->bufs, libxl__datacopier_bufs);
+        if (!buf || buf->used >= sizeof(buf->buf)) {
+            buf = malloc(sizeof(*buf));
+            if (!buf) libxl__alloc_failed(CTX, __func__, 1, sizeof(*buf));
+            buf->used = 0;
+            LIBXL_TAILQ_INSERT_TAIL(&dc->bufs, buf, entry);
+        }
+        int r = read(ev->fd,
+                     buf->buf + buf->used,
+                     sizeof(buf->buf) - buf->used);
+        if (r < 0) {
+            if (errno == EINTR) continue;
+            if (errno == EWOULDBLOCK) break;
+            LOGE(ERROR, "error reading %s during copy of %s",
+                 dc->readwhat, dc->copywhat);
+            datacopier_callback(egc, dc, 0, errno);
+            return;
+        }
+        if (r == 0) {
+            libxl__ev_fd_deregister(gc, &dc->toread);
+            break;
+        }
+        buf->used += r;
+        dc->used += r;
+        assert(buf->used <= sizeof(buf->buf));
+    }
+    datacopier_check_state(egc, dc);
+}
+
+static void datacopier_writable(libxl__egc *egc, libxl__ev_fd *ev,
+                                int fd, short events, short revents) {
+    libxl__datacopier_state *dc = CONTAINER_OF(ev, *dc, towrite);
+    STATE_AO_GC(dc);
+
+    if (revents & ~POLLOUT) {
+        LOG(ERROR, "unexpected poll event 0x%x (should be POLLOUT)"
+            " on %s during copy of %s", revents, dc->writewhat, dc->copywhat);
+        datacopier_callback(egc, dc, -1, 0);
+        return;
+    }
+    assert(revents & POLLOUT);
+    for (;;) {
+        libxl__datacopier_buf *buf = LIBXL_TAILQ_FIRST(&dc->bufs);
+        if (!buf)
+            break;
+        if (!buf->used) {
+            LIBXL_TAILQ_REMOVE(&dc->bufs, buf, entry);
+            free(buf);
+            continue;
+        }
+        int r = write(ev->fd, buf->buf, buf->used);
+        if (r < 0) {
+            if (errno == EINTR) continue;
+            if (errno == EWOULDBLOCK) break;
+            LOGE(ERROR, "error writing to %s during copy of %s",
+                 dc->writewhat, dc->copywhat);
+            datacopier_callback(egc, dc, 1, errno);
+            return;
+        }
+        assert(r > 0);
+        assert(r <= buf->used);
+        buf->used -= r;
+        dc->used -= r;
+        assert(dc->used >= 0);
+        memmove(buf->buf, buf->buf+r, buf->used);
+    }
+    datacopier_check_state(egc, dc);
+}
+
+int libxl__datacopier_start(libxl__datacopier_state *dc)
+{
+    int rc;
+    STATE_AO_GC(dc);
+
+    libxl__datacopier_init(dc);
+
+    rc = libxl__ev_fd_register(gc, &dc->toread, datacopier_readable,
+                               dc->readfd, POLLIN);
+    if (rc) goto out;
+
+    rc = libxl__ev_fd_register(gc, &dc->towrite, datacopier_writable,
+                               dc->writefd, POLLOUT);
+    if (rc) goto out;
+
+    return 0;
+
+ out:
+    libxl__datacopier_kill(dc);
+    return rc;
+}
+
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index a3955a4..8eabcfa 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -830,6 +830,7 @@ _hidden int libxl__ev_devstate_wait(libxl__gc *gc, libxl__ev_devstate *ds,
  */
 _hidden int libxl__try_phy_backend(mode_t st_mode);
 
+
 /* from libxl_pci */
 
 _hidden int libxl__device_pci_add(libxl__gc *gc, uint32_t domid, libxl_device_pci *pcidev, int starting);
@@ -1455,6 +1456,45 @@ int libxl__carefd_close(libxl__carefd*);
 int libxl__carefd_fd(const libxl__carefd*);
 
 
+/*----- datacopier: copies data from one fd to another -----*/
+
+typedef struct libxl__datacopier_state libxl__datacopier_state;
+typedef struct libxl__datacopier_buf libxl__datacopier_buf;
+
+/* onwrite==1 means failure happened when writing, logged, errno is valid
+ * onwrite==0 means failure happened when reading
+ *     errno==0 means we got eof and all data was written
+ *     errno!=0 means we had a read error, logged
+ * onwrite==-1 means some other internal failure, errnoval not valid, logged
+ * in all cases copier is killed before calling this callback */
+typedef void libxl__datacopier_callback(libxl__egc *egc,
+     libxl__datacopier_state *dc, int onwrite, int errnoval);
+
+struct libxl__datacopier_buf {
+    /* private to datacopier */
+    LIBXL_TAILQ_ENTRY(libxl__datacopier_buf) entry;
+    int used;
+    char buf[1000];
+};
+
+struct libxl__datacopier_state {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ao *ao;
+    int readfd, writefd;
+    ssize_t maxsz;
+    const char *copywhat, *readwhat, *writewhat; /* for error msgs */
+    libxl__datacopier_callback *callback;
+    /* remaining fields are private to datacopier */
+    libxl__ev_fd toread, towrite;
+    ssize_t used;
+    LIBXL_TAILQ_HEAD(libxl__datacopier_bufs, libxl__datacopier_buf) bufs;
+};
+
+_hidden void libxl__datacopier_init(libxl__datacopier_state *dc);
+_hidden void libxl__datacopier_kill(libxl__datacopier_state *dc);
+_hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
+
+
 /*
  * Convenience macros.
  */
-- 
1.7.2.5

[PATCH 28/31] libxl: provide libxl__openpty_*

[Ian Jackson]

General facility for ao operations to open ptys.

This will be used by the bootloader machinery.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_aoutils.c  |  127 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   36 ++++++++++++
 2 files changed, 163 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index c1229e4..b33abe4 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -187,3 +187,130 @@ int libxl__datacopier_start(libxl__datacopier_state *dc)
     return rc;
 }
 
+/*----- openpty -----*/
+
+/* implementation */
+    
+static void openpty_cleanup(libxl__openpty_state *op)
+{
+    int i;
+
+    for (i=0; i<op->count; i++) {
+        libxl__openpty_result *res = &op->results[i];
+        libxl__carefd_close(res->master);  res->master = 0;
+        libxl__carefd_close(res->slave);   res->slave = 0;
+    }
+}
+
+static void openpty_exited(libxl__egc *egc, libxl__ev_child *child,
+                           pid_t pid, int status) {
+    libxl__openpty_state *op = CONTAINER_OF(child, *op, child);
+    STATE_AO_GC(op);
+
+    if (status) {
+        /* Perhaps the child gave us the fds and then exited nonzero.
+         * Well that would be odd but we don't really care. */
+        libxl_report_child_exitstatus(CTX, op->rc ? LIBXL__LOG_ERROR
+                                                  : LIBXL__LOG_WARNING,
+                                      "openpty child", pid, status);
+    }
+    if (op->rc)
+        openpty_cleanup(op);
+    op->callback(egc, op);
+}
+
+int libxl__openptys(libxl__openpty_state *op,
+                    const struct termios *termp,
+                    const struct winsize *winp) {
+    /*
+     * This is completely crazy.  openpty calls grantpt which the spec
+     * says may fork, and may not be called with a SIGCHLD handler.
+     * Now our application may have a SIGCHLD handler so that's bad.
+     * We could perhaps block it but we'd need to block it on all
+     * threads.  This is just Too Hard.
+     *
+     * So instead, we run openpty in a child process.  That child
+     * process then of course has only our own thread and our own
+     * signal handlers.  We pass the fds back.
+     *
+     * Since our only current caller actually wants two ptys, we
+     * support calling openpty multiple times for a single fork.
+     */
+    STATE_AO_GC(op);
+    int count = op->count;
+    int r, i, rc, sockets[2], ptyfds[count][2];
+    libxl__carefd *for_child = 0;
+    pid_t pid = -1;
+
+    for (i=0; i<count; i++) {
+        ptyfds[i][0] = ptyfds[i][1] = -1;
+        libxl__openpty_result *res = &op->results[i];
+        assert(!res->master);
+        assert(!res->slave);
+    }
+    sockets[0] = sockets[1] = -1; /* 0 is for us, 1 for our child */
+
+    libxl__carefd_begin();
+    r = socketpair(AF_UNIX, SOCK_STREAM, 0, sockets);
+    if (r) { sockets[0] = sockets[1] = -1; }
+    for_child = libxl__carefd_opened(CTX, sockets[1]);
+    if (r) { LOGE(ERROR,"socketpair failed"); rc = ERROR_FAIL; goto out; }
+
+    pid = libxl__ev_child_fork(gc, &op->child, openpty_exited);
+    if (pid == -1) {
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    if (!pid) {
+        /* child */
+        close(sockets[0]);
+        signal(SIGCHLD, SIG_DFL);
+
+        for (i=0; i<count; i++) {
+            r = openpty(&ptyfds[i][0], &ptyfds[i][1], NULL, termp, winp);
+            if (r) { LOGE(ERROR,"openpty failed"); _exit(-1); }
+        }
+        rc = libxl__sendmsg_fds(gc, sockets[1], "",1,
+                                2*count, &ptyfds[0][0], "ptys");
+        if (rc) { LOGE(ERROR,"sendmsg to parent failed"); _exit(-1); }
+        _exit(0);
+    }
+
+    libxl__carefd_close(for_child);
+    for_child = 0;
+
+    /* this should be fast so do it synchronously */
+
+    libxl__carefd_begin();
+    char buf[1];
+    rc = libxl__recvmsg_fds(gc, sockets[0], buf,1,
+                            2*count, &ptyfds[0][0], "ptys");
+    if (!rc) {
+        for (i=0; i<count; i++) {
+            libxl__openpty_result *res = &op->results[i];
+            res->master = libxl__carefd_record(CTX, ptyfds[i][0]);
+            res->slave =  libxl__carefd_record(CTX, ptyfds[i][1]);
+        }
+    }
+    /* now the pty fds are in the carefds, if they were ever open */
+    libxl__carefd_unlock();
+    if (rc)
+        goto out;
+
+    rc = 0;
+
+ out:
+    if (sockets[0] >= 0) close(sockets[0]);
+    libxl__carefd_close(for_child);
+    if (libxl__ev_child_inuse(&op->child)) {
+        op->rc = rc;
+        /* we will get a callback when the child dies */
+        return 0;
+    }
+
+    assert(rc);
+    openpty_cleanup(op);
+    return rc;
+}
+
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8eabcfa..6850097 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1495,6 +1495,42 @@ _hidden void libxl__datacopier_kill(libxl__datacopier_state *dc);
 _hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
 
 
+/*----- openpty -----*/
+
+/*
+ * opens count (>0) ptys like count calls to openpty, and then
+ * calls back.  On entry, all op[].master and op[].slave must be
+ * 0.  On callback, either rc==0 and master and slave are non-0,
+ * or rc is a libxl error and they are both 0.  If libxl__openpty
+ * returns non-0 no callback will happen and everything is left
+ * cleaned up.
+ */
+
+typedef struct libxl__openpty_state libxl__openpty_state;
+typedef struct libxl__openpty_result libxl__openpty_result;
+typedef void libxl__openpty_callback(libxl__egc *egc, libxl__openpty_state *op);
+
+struct libxl__openpty_state {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ao *ao;
+    libxl__openpty_callback *callback;
+    int count;
+    libxl__openpty_result *results; /* actual size is count, out parameter */
+    /* public, result, caller may only read in callback */
+    int rc;
+    /* private for implementation */
+    libxl__ev_child child;
+};
+
+struct libxl__openpty_result {
+    libxl__carefd *master, *slave;
+};
+
+int libxl__openptys(libxl__openpty_state *op,
+                    const struct termios *termp,
+                    const struct winsize *winp);
+
+
 /*
  * Convenience macros.
  */
-- 
1.7.2.5

[PATCH 29/31] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Convert libxl_run_bootloader to an ao_how-taking function.

It's implemented in terms of libxl__bootloader_run, which can be used
internally.  The resulting code is pretty much a rewrite.

Significant changes include:

- We direct the bootloader's results to a file, not a pipe.  This
  makes it simpler to deal with as we don't have to read it
  concurrently along with everything else.

- We now issue a warning if we find unexpected statements in the
  bootloader results.

- The arrangements for buffering of bootloader input and output
  are completely changed.  Now we have a fixed limit of 64k
  on output, and 4k on input, and discard the oldest data when
  this overflows (which it shouldn't).  There is no timeout
  any more.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.c            |    4 +
 tools/libxl/libxl.h            |    3 +-
 tools/libxl/libxl_bootloader.c |  708 +++++++++++++++++++++-------------------
 tools/libxl/libxl_create.c     |    8 +-
 tools/libxl/libxl_internal.h   |   34 ++
 5 files changed, 415 insertions(+), 342 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 42ac89f..54f3813 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -1748,6 +1748,10 @@ int libxl_device_disk_local_detach(libxl_ctx *ctx, libxl_device_disk *disk)
      * For other device types assume that the blktap2 process is
      * needed by the soon to be started domain and do nothing.
      */
+    /*
+     * FIXME
+     * This appears to leak the disk in failure paths
+     */
 
     return 0;
 }
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 03e71f6..477b72a 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -494,7 +494,8 @@ int libxl_get_max_cpus(libxl_ctx *ctx);
 int libxl_run_bootloader(libxl_ctx *ctx,
                          libxl_domain_build_info *info,
                          libxl_device_disk *disk,
-                         uint32_t domid);
+                         uint32_t domid,
+                         libxl_asyncop_how *ao_how);
 
   /* 0 means ERROR_ENOMEM, which we have logged */
 
diff --git a/tools/libxl/libxl_bootloader.c b/tools/libxl/libxl_bootloader.c
index b50944a..1361117 100644
--- a/tools/libxl/libxl_bootloader.c
+++ b/tools/libxl/libxl_bootloader.c
@@ -15,6 +15,7 @@
 #include "libxl_osdeps.h" /* must come before any other headers */
 
 #include <termios.h>
+#include <utmp.h>
 
 #ifdef INCLUDE_LIBUTIL_H
 #include INCLUDE_LIBUTIL_H
@@ -22,67 +23,80 @@
 
 #include "libxl_internal.h"
 
-#define XENCONSOLED_BUF_SIZE 16
-#define BOOTLOADER_BUF_SIZE 4096
-#define BOOTLOADER_TIMEOUT 1
+#define BOOTLOADER_BUF_OUT 65536
+#define BOOTLOADER_BUF_IN   4096
 
-static char **make_bootloader_args(libxl__gc *gc,
-                                   libxl_domain_build_info *info,
-                                   uint32_t domid,
-                                   const char *fifo, char *disk)
+static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op);
+static void bootloader_keystrokes_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval);
+static void bootloader_display_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval);
+static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
+                                pid_t pid, int status);
+
+/*----- bootloader arguments -----*/
+
+static void bootloader_arg(libxl__bootloader_state *bl, const char *arg)
+{
+    assert(bl->nargs < bl->argsspace);
+    bl->args[bl->nargs++] = arg;
+}
+
+static void make_bootloader_args(libxl__gc *gc, libxl__bootloader_state *bl)
 {
-    flexarray_t *args;
-    int nr = 0;
+    const libxl_domain_build_info *info = bl->info;
 
-    args = flexarray_make(1, 1);
-    if (!args)
-        return NULL;
+    bl->argsspace = 7 + libxl_string_list_length(&info->u.pv.bootloader_args);
 
-    flexarray_set(args, nr++, (char *)info->u.pv.bootloader);
+    GCNEW_ARRAY(bl->args, bl->argsspace);
+
+#define ARG(arg) bootloader_arg(bl, (arg))
+
+    ARG(info->u.pv.bootloader);
 
     if (info->u.pv.kernel.path)
-        flexarray_set(args, nr++, libxl__sprintf(gc, "--kernel=%s",
-                                                 info->u.pv.kernel.path));
+        ARG(libxl__sprintf(gc, "--kernel=%s", info->u.pv.kernel.path));
     if (info->u.pv.ramdisk.path)
-        flexarray_set(args, nr++, libxl__sprintf(gc, "--ramdisk=%s", info->u.pv.ramdisk.path));
+        ARG(libxl__sprintf(gc, "--ramdisk=%s", info->u.pv.ramdisk.path));
     if (info->u.pv.cmdline && *info->u.pv.cmdline != '\0')
-        flexarray_set(args, nr++, libxl__sprintf(gc, "--args=%s", info->u.pv.cmdline));
+        ARG(libxl__sprintf(gc, "--args=%s", info->u.pv.cmdline));
 
-    flexarray_set(args, nr++, libxl__sprintf(gc, "--output=%s", fifo));
-    flexarray_set(args, nr++, "--output-format=simple0");
-    flexarray_set(args, nr++, libxl__sprintf(gc, "--output-directory=%s", "/var/run/libxl/"));
+    ARG(libxl__sprintf(gc, "--output=%s", bl->outputpath));
+    ARG("--output-format=simple0");
+    ARG(libxl__sprintf(gc, "--output-directory=%s", bl->outputdir));
 
     if (info->u.pv.bootloader_args) {
         char **p = info->u.pv.bootloader_args;
         while (*p) {
-            flexarray_set(args, nr++, *p);
+            ARG(*p);
             p++;
         }
     }
 
-    flexarray_set(args, nr++, disk);
+    ARG(bl->diskpath);
 
-    /* Sentinal for execv */
-    flexarray_set(args, nr++, NULL);
+    /* Sentinel for execv */
+    ARG(NULL);
 
-    return (char **) flexarray_contents(args); /* Frees args */
+#undef ARG
 }
 
-static int open_xenconsoled_pty(int *master, int *slave, char *slave_path, size_t slave_path_len)
+/*----- synchronous subroutines -----*/
+
+static int setup_xenconsoled_pty(libxl__egc *egc, libxl__bootloader_state *bl,
+                                 char *slave_path, size_t slave_path_len)
 {
+    STATE_AO_GC(bl);
     struct termios termattr;
-    int ret;
-
-    ret = openpty(master, slave, NULL, NULL, NULL);
-    if (ret < 0)
-        return -1;
-
-    ret = ttyname_r(*slave, slave_path, slave_path_len);
-    if (ret == -1) {
-        close(*master);
-        close(*slave);
-        *master = *slave = -1;
-        return -1;
+    int r, rc;
+    int slave = libxl__carefd_fd(bl->ptys[1].slave);
+    int master = libxl__carefd_fd(bl->ptys[1].master);
+
+    r = ttyname_r(slave, slave_path, slave_path_len);
+    if (r == -1) {
+        LOGE(ERROR,"ttyname_r failed");
+        rc = ERROR_FAIL;
+        goto out;
     }
 
     /*
@@ -95,310 +109,215 @@ static int open_xenconsoled_pty(int *master, int *slave, char *slave_path, size_
      * semantics on Solaris, so don't try to set any attributes
      * for it.
      */
-#if !defined(__sun__) && !defined(__NetBSD__)
-    tcgetattr(*master, &termattr);
+    tcgetattr(master, &termattr);
     cfmakeraw(&termattr);
-    tcsetattr(*master, TCSANOW, &termattr);
-
-    close(*slave);
-    *slave = -1;
-#else
-    tcgetattr(*slave, &termattr);
-    cfmakeraw(&termattr);
-    tcsetattr(*slave, TCSANOW, &termattr);
-#endif
-
-    fcntl(*master, F_SETFL, O_NDELAY);
-    fcntl(*master, F_SETFD, FD_CLOEXEC);
+    tcsetattr(master, TCSANOW, &termattr);
 
     return 0;
+
+ out:
+    return rc;
 }
 
-static pid_t fork_exec_bootloader(int *master, const char *arg0, char **args)
-{
-    struct termios termattr;
-    pid_t pid = forkpty(master, NULL, NULL, NULL);
-    if (pid == -1)
-        return -1;
-    else if (pid == 0) {
-        setenv("TERM", "vt100", 1);
-        libxl__exec(-1, -1, -1, arg0, args);
-        return -1;
-    }
+static const char *bootloader_result_command(libxl__gc *gc, const char *buf,
+                         const char *prefix, size_t prefixlen) {
+    if (strncmp(buf, prefix, prefixlen))
+        return 0;
 
-    /*
-     * On Solaris, the master pty side does not have terminal semantics,
-     * so don't try to set any attributes, as it will fail.
-     */
-#if !defined(__sun__)
-    tcgetattr(*master, &termattr);
-    cfmakeraw(&termattr);
-    tcsetattr(*master, TCSANOW, &termattr);
-#endif
+    const char *rhs = buf + prefixlen;
+    if (!CTYPE(isspace,*rhs))
+        return 0;
+
+    while (CTYPE(isspace,*rhs))
+        rhs++;
 
-    fcntl(*master, F_SETFL, O_NDELAY);
+    LOG(DEBUG,"bootloader output contained %s %s", prefix, rhs);
 
-    return pid;
+    return rhs;
 }
 
-/*
- * filedescriptors:
- *   fifo_fd        - bootstring output from the bootloader
- *   xenconsoled_fd - input/output from/to xenconsole
- *   bootloader_fd  - input/output from/to pty that controls the bootloader
- * The filedescriptors are NDELAY, so it's ok to try to read
- * bigger chunks than may be available, to keep e.g. curses
- * screen redraws in the bootloader efficient. xenconsoled_fd is the side that
- * gets xenconsole input, which will be keystrokes, so a small number
- * is sufficient. bootloader_fd is pygrub output, which will be curses screen
- * updates, so a larger number (1024) is appropriate there.
- *
- * For writeable descriptors, only include them in the set for select
- * if there is actual data to write, otherwise this would loop too fast,
- * eating up CPU time.
- */
-static char * bootloader_interact(libxl__gc *gc, int xenconsoled_fd, int bootloader_fd, int fifo_fd)
+static int parse_bootloader_result(libxl__egc *egc,
+                                   libxl__bootloader_state *bl)
 {
-    int ret;
-
-    size_t nr_out = 0, size_out = 0;
-    char *output = NULL;
-    struct timeval wait;
-
-    /* input from xenconsole. read on xenconsoled_fd write to bootloader_fd */
-    int xenconsoled_prod = 0, xenconsoled_cons = 0;
-    char xenconsoled_buf[XENCONSOLED_BUF_SIZE];
-    /* output from bootloader. read on bootloader_fd write to xenconsoled_fd */
-    int bootloader_prod = 0, bootloader_cons = 0;
-    char bootloader_buf[BOOTLOADER_BUF_SIZE];
-
-    while(1) {
-        fd_set wsel, rsel;
-        int nfds;
-
-        /* Set timeout to 1s before starting to discard data */
-        wait.tv_sec = BOOTLOADER_TIMEOUT;
-        wait.tv_usec = 0;
-
-        /* Move buffers around to drop already consumed data */
-        if (xenconsoled_cons > 0) {
-            xenconsoled_prod -= xenconsoled_cons;
-            memmove(xenconsoled_buf, &xenconsoled_buf[xenconsoled_cons],
-                    xenconsoled_prod);
-            xenconsoled_cons = 0;
-        }
-        if (bootloader_cons > 0) {
-            bootloader_prod -= bootloader_cons;
-            memmove(bootloader_buf, &bootloader_buf[bootloader_cons],
-                    bootloader_prod);
-            bootloader_cons = 0;
-        }
-
-        FD_ZERO(&rsel);
-        FD_SET(fifo_fd, &rsel);
-        nfds = fifo_fd + 1;
-        if (xenconsoled_prod < XENCONSOLED_BUF_SIZE) {
-            /* The buffer is not full, try to read more data */
-            FD_SET(xenconsoled_fd, &rsel);
-            nfds = xenconsoled_fd + 1 > nfds ? xenconsoled_fd + 1 : nfds;
-        } 
-        if (bootloader_prod < BOOTLOADER_BUF_SIZE) {
-            /* The buffer is not full, try to read more data */
-            FD_SET(bootloader_fd, &rsel);
-            nfds = bootloader_fd + 1 > nfds ? bootloader_fd + 1 : nfds;
-        }
-
-        FD_ZERO(&wsel);
-        if (bootloader_prod > 0) {
-            /* The buffer has data to consume */
-            FD_SET(xenconsoled_fd, &wsel);
-            nfds = xenconsoled_fd + 1 > nfds ? xenconsoled_fd + 1 : nfds;
-        }
-        if (xenconsoled_prod > 0) {
-            /* The buffer has data to consume */
-            FD_SET(bootloader_fd, &wsel);
-            nfds = bootloader_fd + 1 > nfds ? bootloader_fd + 1 : nfds;
-        }
-
-        if (xenconsoled_prod == XENCONSOLED_BUF_SIZE ||
-            bootloader_prod == BOOTLOADER_BUF_SIZE)
-            ret = select(nfds, &rsel, &wsel, NULL, &wait);
-        else
-            ret = select(nfds, &rsel, &wsel, NULL, NULL);
-        if (ret < 0) {
-            if (errno == EINTR)
-                continue;
-            goto out_err;
-        }
-
-        /* Input from xenconsole, read xenconsoled_fd, write bootloader_fd */
-        if (ret == 0 && xenconsoled_prod == XENCONSOLED_BUF_SIZE) {
-            /* Drop the buffer */
-            xenconsoled_prod = 0;
-            xenconsoled_cons = 0;
-        } else if (FD_ISSET(xenconsoled_fd, &rsel)) {
-            ret = read(xenconsoled_fd, &xenconsoled_buf[xenconsoled_prod], XENCONSOLED_BUF_SIZE - xenconsoled_prod);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                xenconsoled_prod += ret;
-        }
-        if (FD_ISSET(bootloader_fd, &wsel)) {
-            ret = write(bootloader_fd, &xenconsoled_buf[xenconsoled_cons], xenconsoled_prod - xenconsoled_cons);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                xenconsoled_cons += ret;
-        }
+    STATE_AO_GC(bl);
+    char buf[PATH_MAX*2];
+    FILE *f = 0;
+    int rc = ERROR_FAIL;
+    libxl_domain_build_info *info = bl->info;
+
+    f = fopen(bl->outputpath, "r");
+    if (!f) {
+        LOGE(ERROR,"open bootloader output file %s", bl->outputpath);
+        goto out;
+    }
 
-        /* Input from bootloader, read bootloader_fd, write xenconsoled_fd */
-        if (ret == 0 && bootloader_prod == BOOTLOADER_BUF_SIZE) {
-            /* Drop the buffer */
-            bootloader_prod = 0;
-            bootloader_cons = 0;
-        } else if (FD_ISSET(bootloader_fd, &rsel)) {
-            ret = read(bootloader_fd, &bootloader_buf[bootloader_prod], BOOTLOADER_BUF_SIZE - bootloader_prod);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                bootloader_prod += ret;
-        }
-        if (FD_ISSET(xenconsoled_fd, &wsel)) {
-            ret = write(xenconsoled_fd, &bootloader_buf[bootloader_cons], bootloader_prod - bootloader_cons);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                bootloader_cons += ret;
+    for (;;) {
+        /* Read a nul-terminated "line" and put the result in
+         * buf, and its length (not including the nul) in l */
+        int l = 0, c;
+        while ((c = getc(f)) != EOF && c != '\0') {
+            if (l < sizeof(buf)-1)
+                buf[l] = c;
+            l++;
         }
-
-        if (FD_ISSET(fifo_fd, &rsel)) {
-            if (size_out - nr_out < 256) {
-                char *temp;
-                size_t new_size = size_out == 0 ? 32 : size_out * 2;
-
-                temp = realloc(output, new_size);
-                if (temp == NULL)
-                    goto out_err;
-                output = temp;
-                memset(output + size_out, 0, new_size - size_out);
-                size_out = new_size;
+        if (c == EOF) {
+            if (ferror(f)) {
+                LOGE(ERROR,"read bootloader output file %s", bl->outputpath);
+                goto out;
             }
-
-            ret = read(fifo_fd, output + nr_out, size_out - nr_out);
-            if (ret > 0)
-                  nr_out += ret;
-            if (ret == 0)
+            if (!l)
                 break;
         }
-    }
-
-    libxl__ptr_add(gc, output);
-    return output;
+        if (l >= sizeof(buf)) {
+            LOG(WARN,"bootloader output contained"
+                " overly long item `%.150s...'", buf);
+            continue;
+        }
+        buf[l] = 0;
 
-out_err:
-    free(output);
-    return NULL;
-}
+        const char *rhs;
+#define COMMAND(s) ((rhs = bootloader_result_command(gc, buf, s, sizeof(s)-1)))
 
-static void parse_bootloader_result(libxl__gc *gc,
-                                    libxl_domain_build_info *info,
-                                    const char *o)
-{
-    while (*o != '\0') {
-        if (strncmp("kernel ", o, strlen("kernel ")) == 0) {
+        if (COMMAND("kernel")) {
             free(info->u.pv.kernel.path);
-            info->u.pv.kernel.path = strdup(o + strlen("kernel "));
+            info->u.pv.kernel.path = libxl__strdup(NULL, rhs);
             libxl__file_reference_map(&info->u.pv.kernel);
             unlink(info->u.pv.kernel.path);
-        } else if (strncmp("ramdisk ", o, strlen("ramdisk ")) == 0) {
+        } else if (COMMAND("ramdisk")) {
             free(info->u.pv.ramdisk.path);
-            info->u.pv.ramdisk.path = strdup(o + strlen("ramdisk "));
+            info->u.pv.ramdisk.path = libxl__strdup(NULL, rhs);
             libxl__file_reference_map(&info->u.pv.ramdisk);
             unlink(info->u.pv.ramdisk.path);
-        } else if (strncmp("args ", o, strlen("args ")) == 0) {
+        } else if (COMMAND("args")) {
             free(info->u.pv.cmdline);
-            info->u.pv.cmdline = strdup(o + strlen("args "));
+            info->u.pv.cmdline = libxl__strdup(NULL, rhs);
+        } else if (l) {
+            LOG(WARN, "unexpected output from bootloader: `%s'", buf);
         }
-
-        o = o + strlen(o) + 1;
     }
+    rc = 0;
+
+ out:
+    if (f) fclose(f);
+    return rc;
 }
 
-int libxl_run_bootloader(libxl_ctx *ctx,
-                         libxl_domain_build_info *info,
-                         libxl_device_disk *disk,
-                         uint32_t domid)
+
+/*----- init and cleanup -----*/
+
+void libxl__bootloader_init(libxl__bootloader_state *bl)
+{
+    bl->diskpath = NULL;
+    bl->ptys[0].master = bl->ptys[0].slave = 0;
+    bl->ptys[1].master = bl->ptys[1].slave = 0;
+    libxl__ev_child_init(&bl->child);
+    libxl__datacopier_init(&bl->keystrokes);
+    libxl__datacopier_init(&bl->display);
+}
+
+static void bootloader_cleanup(libxl__egc *egc, libxl__bootloader_state *bl)
 {
-    GC_INIT(ctx);
-    int ret, rc = 0;
-    char *fifo = NULL;
-    char *diskpath = NULL;
-    char **args = NULL;
+    STATE_AO_GC(bl);
+    int i;
 
-    char tempdir_template[] = "/var/run/libxl/bl.XXXXXX";
-    char *tempdir;
+    if (bl->outputpath) libxl__remove_file(gc, bl->outputpath);
+    if (bl->outputdir) libxl__remove_directory(gc, bl->outputdir);
 
-    char *dom_console_xs_path;
-    char dom_console_slave_tty_path[PATH_MAX];
+    if (bl->diskpath) {
+        libxl_device_disk_local_detach(CTX, bl->disk);
+        free(bl->diskpath);
+        bl->diskpath = 0;
+    }
+    libxl__datacopier_kill(&bl->keystrokes);
+    libxl__datacopier_kill(&bl->display);
+    for (i=0; i<2; i++) {
+        libxl__carefd_close(bl->ptys[i].master);
+        libxl__carefd_close(bl->ptys[i].slave);
+    }
+}
+
+static void bootloader_setpaths(libxl__gc *gc, libxl__bootloader_state *bl)
+{
+    uint32_t domid = bl->domid;
+    bl->outputdir = GCSPRINTF(XEN_RUN_DIR "/bootloader.%"PRIu32".d", domid);
+    bl->outputpath = GCSPRINTF(XEN_RUN_DIR "/bootloader.%"PRIu32".out", domid);
+}
 
-    int xenconsoled_fd = -1, xenconsoled_slave = -1;
-    int bootloader_fd = -1, fifo_fd = -1;
+static void bootloader_callback(libxl__egc *egc, libxl__bootloader_state *bl,
+                                int rc)
+{
+    bootloader_cleanup(egc, bl);
+    bl->callback(egc, bl, rc);
+}
 
-    int blrc;
-    pid_t pid;
-    char *blout;
+/*----- main flow of control -----*/
 
-    struct stat st_buf;
+void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
+{
+    STATE_AO_GC(bl);
+    libxl_domain_build_info *info = bl->info;
+    int rc, r;
 
-    rc = libxl__domain_build_info_setdefault(gc, info);
-    if (rc) goto out;
+    libxl__bootloader_init(bl);
 
-    if (info->type != LIBXL_DOMAIN_TYPE_PV || !info->u.pv.bootloader)
+    if (info->type != LIBXL_DOMAIN_TYPE_PV || !info->u.pv.bootloader) {
+        bl->callback(egc, bl, 0);
+        rc = 0;
         goto out;
+    }
 
-    rc = libxl__domain_build_info_setdefault(gc, info);
-    if (rc) goto out;
+    bootloader_setpaths(gc, bl);
 
-    rc = ERROR_INVAL;
-    if (!disk)
+    for (;;) {
+        r = mkdir(bl->outputdir, 0600);
+        if (!r) break;
+        if (errno == EINTR) continue;
+        if (errno == EEXIST) break;
+        LOGE(ERROR, "failed to create bootloader dir %s", bl->outputdir);
+        rc = ERROR_FAIL;
         goto out;
+    }
 
-    rc = ERROR_FAIL;
-    ret = mkdir("/var/run/libxl/", S_IRWXU);
-    if (ret < 0 && errno != EEXIST)
+    for (;;) {
+        r = open(bl->outputpath, O_WRONLY|O_CREAT|O_TRUNC, 0600);
+        if (r>=0) { close(r); break; }
+        if (errno == EINTR) continue;
+        LOGE(ERROR, "failed to precreate bootloader output %s", bl->outputpath);
+        rc = ERROR_FAIL;
         goto out;
+    }
 
-    ret = stat("/var/run/libxl/", &st_buf);
-    if (ret < 0)
+    bl->diskpath = libxl_device_disk_local_attach(CTX, bl->disk);
+    if (!bl->diskpath) {
+        rc = ERROR_FAIL;
         goto out;
+    }
 
-    if (!S_ISDIR(st_buf.st_mode))
-        goto out;
+    make_bootloader_args(gc, bl);
 
-    tempdir = mkdtemp(tempdir_template);
-    if (tempdir == NULL)
-        goto out;
+    bl->openpty.ao = ao;
+    bl->openpty.callback = bootloader_gotptys;
+    bl->openpty.count = 2;
+    bl->openpty.results = bl->ptys;
+    rc = libxl__openptys(&bl->openpty, 0,0);
+    if (rc) goto out;
 
-    ret = asprintf(&fifo, "%s/fifo", tempdir);
-    if (ret < 0) {
-        fifo = NULL;
-        goto out_close;
-    }
+    return;
 
-    ret = mkfifo(fifo, 0600);
-    if (ret < 0) {
-        goto out_close;
-    }
+ out:
+    assert(rc);
+    bootloader_callback(egc, bl, rc);
+}
 
-    diskpath = libxl_device_disk_local_attach(ctx, disk);
-    if (!diskpath) {
-        goto out_close;
-    }
+static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op)
+{
+    libxl__bootloader_state *bl = CONTAINER_OF(op, *bl, openpty);
+    STATE_AO_GC(bl);
+    int rc, r;
 
-    args = make_bootloader_args(gc, info, domid, fifo, diskpath);
-    if (args == NULL) {
-        rc = ERROR_NOMEM;
-        goto out_close;
+    if (bl->openpty.rc) {
+        rc = bl->openpty.rc;
+        goto out;
     }
 
     /*
@@ -411,76 +330,187 @@ int libxl_run_bootloader(libxl_ctx *ctx,
      * where we copy characters between the two master fds, as well as
      * listening on the bootloader's fifo for the results.
      */
-    ret = open_xenconsoled_pty(&xenconsoled_fd, &xenconsoled_slave,
+
+    char *dom_console_xs_path;
+    char dom_console_slave_tty_path[PATH_MAX];
+    rc = setup_xenconsoled_pty(egc, bl,
                                &dom_console_slave_tty_path[0],
                                sizeof(dom_console_slave_tty_path));
-    if (ret < 0) {
-        goto out_close;
+    if (rc) goto out;
+
+    char *dompath = libxl__xs_get_dompath(gc, bl->domid);
+    if (!dompath) { rc = ERROR_FAIL; goto out; }
+
+    dom_console_xs_path = GCSPRINTF("%s/console/tty", dompath);
+
+    rc = libxl__xs_write(gc, XBT_NULL, dom_console_xs_path, "%s",
+                         dom_console_slave_tty_path);
+    if (rc) {
+        LOGE(ERROR,"xs write console path %s := %s failed",
+             dom_console_xs_path, dom_console_slave_tty_path);
+        rc = ERROR_FAIL;
+        goto out;
     }
 
-    dom_console_xs_path = libxl__sprintf(gc, "%s/console/tty", libxl__xs_get_dompath(gc, domid));
-    libxl__xs_write(gc, XBT_NULL, dom_console_xs_path, "%s", dom_console_slave_tty_path);
+    int bootloader_master = libxl__carefd_fd(bl->ptys[0].master);
+    int xenconsole_master = libxl__carefd_fd(bl->ptys[1].master);
+
+    libxl_fd_set_nonblock(CTX, bootloader_master, 1);
+    libxl_fd_set_nonblock(CTX, xenconsole_master, 1);
+
+    bl->keystrokes.writefd   = bl->display.readfd   = bootloader_master;
+    bl->keystrokes.writewhat = bl->display.readwhat = "bootloader pty";
+
+    bl->keystrokes.readfd   = bl->display.writefd   = xenconsole_master;
+    bl->keystrokes.readwhat = bl->display.writewhat = "xenconsole client pty";
+
+    bl->keystrokes.ao = ao;
+    bl->keystrokes.maxsz = BOOTLOADER_BUF_OUT;
+    bl->keystrokes.copywhat =
+        GCSPRINTF("bootloader input for domain %"PRIu32, bl->domid);
+    bl->keystrokes.callback = bootloader_keystrokes_copyfail;
+    rc = libxl__datacopier_start(&bl->keystrokes);
+    if (rc) goto out;
+
+    bl->display.ao = ao;
+    bl->display.maxsz = BOOTLOADER_BUF_IN;
+    bl->display.copywhat =
+        GCSPRINTF("bootloader output for domain %"PRIu32, bl->domid);
+    bl->display.callback = bootloader_display_copyfail;
+    rc = libxl__datacopier_start(&bl->display);
+    if (rc) goto out;
+
+    LOG(DEBUG, "executing bootloader: %s", bl->args[0]);
+    for (const char **blarg = bl->args;
+         *blarg;
+         blarg++)
+        LOG(DEBUG, "  bootloader arg: %s", *blarg);
+
+    struct termios termattr;
 
-    pid = fork_exec_bootloader(&bootloader_fd, info->u.pv.bootloader, args);
-    if (pid < 0) {
-        goto out_close;
+    pid_t pid = libxl__ev_child_fork(gc, &bl->child, bootloader_finished);
+    if (pid == -1) {
+        rc = ERROR_FAIL;
+        goto out;
     }
 
-    while (1) {
-        if (waitpid(pid, &blrc, WNOHANG) == pid)
-            goto out_close;
+    if (!pid) {
+        /* child */
+        r = login_tty(libxl__carefd_fd(bl->ptys[0].slave));
+        if (r) { LOGE(ERROR, "login_tty failed"); exit(-1); }
+        setenv("TERM", "vt100", 1);
+        libxl__exec(-1, -1, -1, bl->args[0], (char**)bl->args);
+        exit(-1);
+    }
 
-        fifo_fd = open(fifo, O_RDONLY);
-        if (fifo_fd > -1)
-            break;
+    /* parent */
 
-        if (errno == EINTR)
-            continue;
+    /*
+     * On Solaris, the master pty side does not have terminal semantics,
+     * so don't try to set any attributes, as it will fail.
+     */
+#if !defined(__sun__)
+    tcgetattr(bootloader_master, &termattr);
+    cfmakeraw(&termattr);
+    tcsetattr(bootloader_master, TCSANOW, &termattr);
+#endif
 
-        goto out_close;
+    return;
+
+ out:
+    bootloader_callback(egc, bl, rc);
+}
+
+/* perhaps one of these will be called, but perhaps not */
+static void bootloader_copyfail(libxl__egc *egc, const char *which,
+       libxl__bootloader_state *bl, int onwrite, int errnoval)
+{
+    STATE_AO_GC(bl);
+    int r;
+
+    if (!onwrite && !errnoval)
+        LOG(ERROR, "unexpected eof copying %s", which);
+    libxl__datacopier_kill(&bl->keystrokes);
+    libxl__datacopier_kill(&bl->display);
+    if (bl->child.pid >= 0) {
+        r = kill(bl->child.pid, SIGTERM);
+        if (r) LOGE(WARN, "after failure, failed to kill bootloader [%lu]",
+                    (unsigned long)bl->child.pid);
     }
+    bl->rc = ERROR_FAIL;
+}
+static void bootloader_keystrokes_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval)
+{
+    libxl__bootloader_state *bl = CONTAINER_OF(dc, *bl, keystrokes);
+    bootloader_copyfail(egc, "bootloader input", bl, onwrite, errnoval);
+}
+static void bootloader_display_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval)
+{
+    libxl__bootloader_state *bl = CONTAINER_OF(dc, *bl, display);
+    bootloader_copyfail(egc, "bootloader output", bl, onwrite, errnoval);
+}
 
-    fcntl(fifo_fd, F_SETFL, O_NDELAY);
+static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
+                                pid_t pid, int status) {
+    libxl__bootloader_state *bl = CONTAINER_OF(child, *bl, child);
+    STATE_AO_GC(bl);
+    int rc;
 
-    blout = bootloader_interact(gc, xenconsoled_fd, bootloader_fd, fifo_fd);
-    if (blout == NULL) {
-        goto out_close;
+    libxl__datacopier_kill(&bl->keystrokes);
+    libxl__datacopier_kill(&bl->display);
+
+    if (status) {
+        libxl_report_child_exitstatus(CTX, XTL_ERROR, "bootloader",
+                                      pid, status);
+        rc = ERROR_FAIL;
+        goto out;
+    } else {
+        LOG(DEBUG, "bootloader completed");
     }
 
-    pid = waitpid(pid, &blrc, 0);
-    if (pid == -1 || (pid > 0 && WIFEXITED(blrc) && WEXITSTATUS(blrc) != 0)) {
-        goto out_close;
+    if (bl->rc) {
+        /* datacopier went wrong */
+        rc = bl->rc;
+        goto out;
     }
 
-    parse_bootloader_result(gc, info, blout);
+    rc = parse_bootloader_result(egc, bl);
+    if (rc) goto out;
 
     rc = 0;
-out_close:
-    if (diskpath) {
-        libxl_device_disk_local_detach(ctx, disk);
-        free(diskpath);
-    }
-    if (fifo_fd > -1)
-        close(fifo_fd);
-    if (bootloader_fd > -1)
-        close(bootloader_fd);
-    if (xenconsoled_fd > -1)
-        close(xenconsoled_fd);
-    if (xenconsoled_slave > -1)
-        close(xenconsoled_slave);
-
-    if (fifo) {
-        unlink(fifo);
-        free(fifo);
-    }
+    LOG(DEBUG, "bootloader execution successful");
 
-    rmdir(tempdir);
+ out:
+    bootloader_callback(egc, bl, rc);
+}
 
-    free(args);
+/*----- entrypoint for external callers -----*/
 
-out:
-    GC_FREE;
-    return rc;
+static void run_bootloader_done(libxl__egc *egc,
+                                libxl__bootloader_state *st, int rc)
+{
+    libxl__ao_complete(egc, st->ao, rc);
+}
+
+int libxl_run_bootloader(libxl_ctx *ctx,
+                         libxl_domain_build_info *info,
+                         libxl_device_disk *disk,
+                         uint32_t domid,
+                         libxl_asyncop_how *ao_how)
+{
+    AO_CREATE(ctx,domid,ao_how);
+    libxl__bootloader_state *bl;
+
+    GCNEW(bl);
+    bl->ao = ao;
+    bl->callback = run_bootloader_done;
+    bl->info = info;
+    bl->disk = disk;
+    bl->domid = domid;
+    libxl__bootloader_run(egc, bl);
+    return AO_INPROGRESS;
 }
 
 /*
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 3675afe..dbc3cf0 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -572,8 +572,12 @@ static int do_domain_create(libxl__gc *gc, libxl_domain_config *d_config,
         if (ret) goto error_out;
     }
 
-    if ( restore_fd < 0 ) {
-        ret = libxl_run_bootloader(ctx, &d_config->b_info, d_config->num_disks > 0 ? &d_config->disks[0] : NULL, domid);
+    libxl_device_disk *bootdisk =
+        d_config->num_disks > 0 ? &d_config->disks[0] : NULL;
+
+    if (restore_fd < 0 && bootdisk) {
+        ret = libxl_run_bootloader(ctx, &d_config->b_info, bootdisk, domid,
+                                   0 /* fixme-ao */);
         if (ret) {
             LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
                        "failed to run bootloader: %d", ret);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 6850097..fd96b31 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -42,6 +42,7 @@
 #include <sys/time.h>
 #include <sys/types.h>
 #include <sys/wait.h>
+#include <sys/socket.h>
 
 #include <xs.h>
 #include <xenctrl.h>
@@ -449,6 +450,7 @@ _hidden int libxl__remove_file(libxl__gc *gc, const char *path);
 _hidden int libxl__remove_directory(libxl__gc *gc, const char *path);
 _hidden int libxl__remove_file_or_directory(libxl__gc *gc, const char *path);
 
+
 _hidden char **libxl__xs_kvs_of_flexarray(libxl__gc *gc, flexarray_t *array, int length);
 
 _hidden int libxl__xs_writev(libxl__gc *gc, xs_transaction_t t,
@@ -1531,6 +1533,38 @@ int libxl__openptys(libxl__openpty_state *op,
                     const struct winsize *winp);
 
 
+/*----- bootloader -----*/
+
+typedef struct libxl__bootloader_state libxl__bootloader_state;
+typedef void libxl__run_bootloader_callback(libxl__egc*,
+                                libxl__bootloader_state*, int rc);
+
+struct libxl__bootloader_state {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ao *ao;
+    libxl__run_bootloader_callback *callback;
+    libxl_domain_build_info *info; /* u.pv.{kernel,ramdisk,cmdline} updated */
+    libxl_device_disk *disk;
+    uint32_t domid;
+    /* private to libxl__run_bootloader */
+    char *outputpath, *outputdir;
+    char *diskpath; /* not from gc, represents actually attached disk */
+    libxl__openpty_state openpty;
+    libxl__openpty_result ptys[2];  /* [0] is for bootloader */
+    libxl__ev_child child;
+    int nargs, argsspace;
+    const char **args;
+    libxl__datacopier_state keystrokes, display;
+    int rc;
+};
+
+_hidden void libxl__bootloader_init(libxl__bootloader_state *bl);
+
+/* Will definitely call st->callback, perhaps reentrantly.
+ * If callback is passed rc==0, will have updated st->info appropriately */
+_hidden void libxl__bootloader_run(libxl__egc*, libxl__bootloader_state *st);
+
+
 /*
  * Convenience macros.
  */
-- 
1.7.2.5

[PATCH 30/31] libxl: make libxl_create_logfile const-correct

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_utils.c |    2 +-
 tools/libxl/libxl_utils.h |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index dc25ab0..8eacebf 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -193,7 +193,7 @@ static int logrename(libxl__gc *gc, const char *old, const char *new)
     return 0;
 }
 
-int libxl_create_logfile(libxl_ctx *ctx, char *name, char **full_name)
+int libxl_create_logfile(libxl_ctx *ctx, const char *name, char **full_name)
 {
     GC_INIT(ctx);
     struct stat stat_buf;
diff --git a/tools/libxl/libxl_utils.h b/tools/libxl/libxl_utils.h
index ca53a8a..2b47622 100644
--- a/tools/libxl/libxl_utils.h
+++ b/tools/libxl/libxl_utils.h
@@ -26,7 +26,7 @@ int libxl_name_to_cpupoolid(libxl_ctx *ctx, const char *name, uint32_t *poolid);
 char *libxl_cpupoolid_to_name(libxl_ctx *ctx, uint32_t poolid);
 int libxl_get_stubdom_id(libxl_ctx *ctx, int guest_domid);
 int libxl_is_stubdom(libxl_ctx *ctx, uint32_t domid, uint32_t *target_domid);
-int libxl_create_logfile(libxl_ctx *ctx, char *name, char **full_name);
+int libxl_create_logfile(libxl_ctx *ctx, const char *name, char **full_name);
 int libxl_string_to_backend(libxl_ctx *ctx, char *s, libxl_disk_backend *backend);
 
 int libxl_read_file_contents(libxl_ctx *ctx, const char *filename,
-- 
1.7.2.5

[PATCH 31/31] libxl: log bootloader output

[Ian Jackson]

This involves adding a new log feature to libxl__datacopier, and then
using it.

If the bootloader exits nonzero we print the log filename in a log
message from libxl.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_aoutils.c    |   10 ++++++++++
 tools/libxl/libxl_bootloader.c |   24 ++++++++++++++++++++++++
 tools/libxl/libxl_internal.h   |    3 ++-
 3 files changed, 36 insertions(+), 1 deletions(-)

diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index b33abe4..1b17c84 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -118,6 +118,16 @@ static void datacopier_readable(libxl__egc *egc, libxl__ev_fd *ev,
             libxl__ev_fd_deregister(gc, &dc->toread);
             break;
         }
+        if (dc->log) {
+            int wrote = fwrite(buf->buf + buf->used, 1, r, dc->log);
+            if (wrote != r) {
+                assert(ferror(dc->log));
+                assert(errno);
+                LOGE(ERROR, "error logging %s", dc->copywhat);
+                datacopier_callback(egc, dc, 0, errno);
+                return;
+            }
+        }
         buf->used += r;
         dc->used += r;
         assert(buf->used <= sizeof(buf->buf));
diff --git a/tools/libxl/libxl_bootloader.c b/tools/libxl/libxl_bootloader.c
index 1361117..47ced8a 100644
--- a/tools/libxl/libxl_bootloader.c
+++ b/tools/libxl/libxl_bootloader.c
@@ -234,6 +234,10 @@ static void bootloader_cleanup(libxl__egc *egc, libxl__bootloader_state *bl)
         libxl__carefd_close(bl->ptys[i].master);
         libxl__carefd_close(bl->ptys[i].slave);
     }
+    if (bl->display.log) {
+        fclose(bl->display.log);
+        bl->display.log = NULL;
+    }
 }
 
 static void bootloader_setpaths(libxl__gc *gc, libxl__bootloader_state *bl)
@@ -256,6 +260,8 @@ void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
 {
     STATE_AO_GC(bl);
     libxl_domain_build_info *info = bl->info;
+    uint32_t domid = bl->domid;
+    char *logfile_tmp = NULL;
     int rc, r;
 
     libxl__bootloader_init(bl);
@@ -268,6 +274,22 @@ void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
 
     bootloader_setpaths(gc, bl);
 
+    const char *logfile_leaf = GCSPRINTF("bootloader.%"PRIu32, domid);
+    rc = libxl_create_logfile(CTX, logfile_leaf, &logfile_tmp);
+    if (rc) goto out;
+
+    /* Transfer ownership of log filename to bl and the gc */
+    bl->logfile = logfile_tmp;
+    libxl__ptr_add(gc, logfile_tmp);
+    logfile_tmp = NULL;
+
+    bl->display.log = fopen(bl->logfile, "a");
+    if (!bl->display.log) {
+        LOGE(ERROR, "failed to create bootloader logfile %s", bl->logfile);
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
     for (;;) {
         r = mkdir(bl->outputdir, 0600);
         if (!r) break;
@@ -305,6 +327,7 @@ void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
     return;
 
  out:
+    free(logfile_tmp);
     assert(rc);
     bootloader_callback(egc, bl, rc);
 }
@@ -462,6 +485,7 @@ static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
     libxl__datacopier_kill(&bl->display);
 
     if (status) {
+        LOG(ERROR, "bootloader failed - consult logfile %s", bl->logfile);
         libxl_report_child_exitstatus(CTX, XTL_ERROR, "bootloader",
                                       pid, status);
         rc = ERROR_FAIL;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index fd96b31..c846144 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1485,6 +1485,7 @@ struct libxl__datacopier_state {
     int readfd, writefd;
     ssize_t maxsz;
     const char *copywhat, *readwhat, *writewhat; /* for error msgs */
+    FILE *log; /* gets a copy of everything */
     libxl__datacopier_callback *callback;
     /* remaining fields are private to datacopier */
     libxl__ev_fd toread, towrite;
@@ -1547,7 +1548,7 @@ struct libxl__bootloader_state {
     libxl_device_disk *disk;
     uint32_t domid;
     /* private to libxl__run_bootloader */
-    char *outputpath, *outputdir;
+    char *outputpath, *outputdir, *logfile;
     char *diskpath; /* not from gc, represents actually attached disk */
     libxl__openpty_state openpty;
     libxl__openpty_result ptys[2];  /* [0] is for bootloader */
-- 
1.7.2.5

Re: [PATCH 01/31] .gitignore: Add a missing file

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  .gitignore |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/.gitignore b/.gitignore
> index 2782ee5..cd12b56 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -357,6 +357,7 @@ tools/firmware/etherboot/eb-roms.h
>  tools/firmware/etherboot/gpxe-git-snapshot.tar.gz
>  tools/misc/xenwatchdogd
>  tools/misc/xen-hvmcrash
> +tools/misc/xen-lowmemd
>  tools/libvchan/vchan-node[12]
>  tools/ocaml/*/.ocamldep.make
>  tools/ocaml/*/*.cm[ixao]

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> Formally change the libxl memory allocation failure policy to "crash".
> 
> Previously we had a very uneven approach; much code assumed that
> libxl__sprintf (for example) would never return NULL, but some code
> was written more carefully.
> 
> We think it is unlikely that we will be able to make the library
> actually robust against allocation failure (since that would be an
> awful lot of never-tested error paths) and few calling environments
> will be able to cope anyway.  So, instead, adopt the alternative
> approach: provide allocation functions which never return null, but
> will crash the whole process instead.
> 
> Consequently,
>  - New noreturn function libxl__alloc_failed which may be used for
>    printing a vaguely-useful error message, rather than simply
>    dereferencing a null pointer.

We got that in the next patch?

>  - libxl__ptr_add now returns void as it crashes on failure.
>  - libxl__zalloc, _calloc, _strdup, _strndup, crash on failure using
>    libxl__alloc_failed.  So all the code that uses these can no longer
>    dereference null on malloc failure.
> 
> While we're at it, make libxl__ptr_add use realloc rather than
> emulating it with calloc and free, and make it grow the array
> exponentially rather than linearly.
> 
> Things left to do:
>  - Provide versions of malloc, realloc and free which do the
>    checking but which do not enroll the pointer in the gc.
>  - Remove a lot of now-spurious error handling.
>  - Remove the ERROR_NOMEM error code.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Couple of unimportant nits below.

>              gc->alloc_ptrs[i] = ptr;
> -            return 0;
> +            return;
>          }
>      }
> -    /* realloc alloc_ptrs manually with calloc/free/replace */
> -    re = calloc(gc->alloc_maxsize + 25, sizeof(void *));
> -    if (!re)
> -        return -1;
> -    for (i = 0; i < gc->alloc_maxsize; i++)
> -        re[i] = gc->alloc_ptrs[i];
> -    /* assign the next pointer */
> -    re[i] = ptr;
> +    int new_maxsize = gc->alloc_maxsize * 2 + 25;

+ 25? (I don't mind, seems even more arbitrary now that we have the *2
though).

> +    assert(new_maxsize < INT_MAX / sizeof(void*) / 2);
> +    gc->alloc_ptrs = realloc(gc->alloc_ptrs, new_maxsize * sizeof(void *));
> +    if (!gc->alloc_ptrs)
> +        libxl__alloc_failed(CTX, __func__, sizeof(void*), new_maxsize);

Strictly this should be "..., new_maxsize, sizeof(void*)" since the
arguments are nmemb and size?

Ian.

Re: [PATCH v4 00/31] libxl child process handling

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> This series has now been tested and is in a suitable form for review
> and, if thought fit, committing to xen-unstable.  It still culminates
> in a patch to convert bootloader execution to the new event machinery.
> 
> Changes since v3 ("[RFC PATCH 00/20]") include:
>  - All comments on previous version addressed.
>  - Several new bugfix patches etc., marked with * below.
>  - Broke the bootloader execution rewrite up into three patches.
>  - Testing and consequent fixes (only to previously un-acked patches);
>    in particular, the final patch has some significant changes.
> 
> Bugfixes for problems reported by Roger Pau Monne:
>   02/31 libxl: ao: allow immediate completion
>   03/31 libxl: fix hang due to libxl__initiate_device_remove
>   04/31 libxl: Fix eventloop_iteration over-locking
> * 05/31 libxl: remove poller from list in libxl__poller_get
> 
> Other general bugfixes:
> * 01/31 .gitignore: Add a missing file
>   06/31 libxl: Fix leak of ctx->lock
>   07/31 tools: Correct PTHREAD options in config/StdGNU.mk
>   08/31 libxl: Use PTHREAD_CFLAGS, LDFLAGS, LIBS
>   09/31 tools: Use PTHREAD_CFLAGS, _LDFLAGS, _LIBS

I think all of the above are now acked and since they are largely
unrelated to the overall thrust of the series could easily go in now?

>   26/31 libxl: Clean up setdefault in do_domain_create
>   30/31 libxl: make libxl_create_logfile const-correct
> 
> Clarifications and improvements related to memory allocation:
>   10/31 libxl: Crash (more sensibly) on malloc failure
>   11/31 libxl: Make libxl__zalloc et al tolerate a NULL gc

These two could probably go in now too, especially 10/31 would be useful
to have sooner rather than later so we can stop writing error handling
code...

Actually the more I read the more I see that a good chunk of the front
of this series could go in now. Is that what you are planning to do?

> Preparatory work:
>   12/31 libxl: Introduce some convenience macros
>   13/31 libxl: include <ctype.h> and introduce CTYPE helper macro
>   14/31 libxl: Provide libxl_string_list_length
>   15/31 libxl: include <_libxl_paths.h> in libxl_internal.h
>   16/31 libxl: abolish libxl_ctx_postfork
> * 23/31 autoconf: New test for openpty et al.
> * 24/31 libxl: provide libxl__remove_file et al.
> * 25/31 libxl: Introduce libxl__sendmsg_fds and libxl__recvmsg_fds
> 
> Event-related infrastructure and fixes:
>   17/31 libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS
>   18/31 libxl: Protect fds with CLOEXEC even with forking threads
> * 19/31 libxl: provide STATE_AO_GC
> * 20/31 libxl: handle POLLERR, POLLHUP, POLLNVAL properly
> * 21/31 libxl: support multiple libxl__ev_fds for the same fd
>   22/31 libxl: event API: new facilities for waiting for subprocesses
>   27/31 libxl: provide libxl__datacopier_*
>   28/31 libxl: provide libxl__openpty_*
>   29/31 libxl: ao: Convert libxl_run_bootloader
> * 31/31 libxl: log bootloader output
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH 18/31] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> We introduce a new "carefd" concept, which relates to fds that we care
> about not being inherited by long-lived children.
> 
> As yet we do not use this anywhere in libxl.  Until all locations in
> libxl which make such fds are converted, libxl__postfork may not work
> entirely properly.  If these locations do not use O_CLOEXEC (or use
> calls for which there is no O_CLOEXEC) then multithreaded programs may
> not work properly.
> 
> This introduces a new API call libxl_postfork_child_noexec which must
> be called by applications which make long-running non-execing
> children.  Add the appropriate call to xl's postfork function.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> ---
>  tools/libxl/Makefile         |    2 +-
>  tools/libxl/libxl.c          |    3 +
>  tools/libxl/libxl_event.h    |   13 ++++
>  tools/libxl/libxl_fork.c     |  146 ++++++++++++++++++++++++++++++++++++++++++
>  tools/libxl/libxl_internal.h |   63 ++++++++++++++++++
>  tools/libxl/xl.c             |    3 +
>  6 files changed, 229 insertions(+), 1 deletions(-)
>  create mode 100644 tools/libxl/libxl_fork.c
> 
> diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
> index 9f7e454..e5ea867 100644
> --- a/tools/libxl/Makefile
> +++ b/tools/libxl/Makefile
> @@ -53,7 +53,7 @@ LIBXL_LIBS += -lyajl
>  LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
>                         libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
>                         libxl_internal.o libxl_utils.o libxl_uuid.o libxl_json.o \
> -                       libxl_qmp.o libxl_event.o $(LIBXL_OBJS-y)
> +                       libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
>  LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
> 
>  $(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include $(XEN_ROOT)/tools/config.h
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 8910420..60dbfdc 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -68,6 +68,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
> 
>      /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
> 
> +    rc = libxl__atfork_init(ctx);
> +    if (rc) goto out;
> +
>      rc = libxl__poller_init(ctx, &ctx->poller_app);
>      if (rc) goto out;
> 
> diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
> index ea553f6..2d2196f 100644
> --- a/tools/libxl/libxl_event.h
> +++ b/tools/libxl/libxl_event.h
> @@ -371,6 +371,19 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
>   */
>  void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
> 
> +
> +/*
> + * An application which initialises a libxl_ctx in a parent process
> + * and then forks a child which does not quickly exec, must
> + * instead libxl_postfork_child_noexec in the child.  One call
> + * on any existing (or specially made) ctx is sufficient; after
> + * this all previously existing libxl_ctx's are invalidated and
> + * must not be used - or even freed.  It is harmless to call this
> + * postfork function and then exec anyway.
> + */
> +void libxl_postfork_child_noexec(libxl_ctx *ctx);
> +
> +
>  #endif
> 
>  /*
> diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
> new file mode 100644
> index 0000000..4751ef4
> --- /dev/null
> +++ b/tools/libxl/libxl_fork.c
> @@ -0,0 +1,146 @@
> +/*
> + * Copyright (C) 2012      Citrix Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2.1 only. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + */
> +/*
> + * Internal child process machinery for use by other parts of libxl
> + */
> +
> +#include "libxl_osdeps.h" /* must come before any other headers */
> +
> +#include "libxl_internal.h"
> +
> +/*
> + * carefd arrangements
> + *
> + * carefd_begin and _unlock take out the no_forking lock, which we
> + * also take and release in our pthread_atfork handlers.  So when this
> + * lock is held the whole process cannot fork.  We therefore protect
> + * our fds from leaking into children made by other threads.
> + *
> + * We maintain a list of all the carefds, so that if the application
> + * wants to fork a long-running but non-execing child, we can close
> + * them all.
> + *
> + * So the record function sets CLOEXEC for the benefit of execing
> + * children, and makes a note of the fd for the benefit of non-execing
> + * ones.
> + */
> +
> +struct libxl__carefd {
> +    LIBXL_LIST_ENTRY(libxl__carefd) entry;
> +    int fd;
> +};
> +
> +static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;
> +static int atfork_registered;
> +static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
> +    LIBXL_LIST_HEAD_INITIALIZER(carefds);
> +
> +static void atfork_lock(void)
> +{
> +    int r = pthread_mutex_lock(&no_forking);
> +    assert(!r);
> +}
> +
> +static void atfork_unlock(void)
> +{
> +    int r = pthread_mutex_unlock(&no_forking);
> +    assert(!r);
> +}
> +
> +int libxl__atfork_init(libxl_ctx *ctx)
> +{
> +    int r, rc;
> +
> +    atfork_lock();
> +    if (atfork_registered) { rc = 0; goto out; }
> +
> +    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
> +    if (r) libxl__alloc_failed(ctx, __func__, 0,0);

This is a bit subtle -- the only documented error from pthread_atfork is
ENOMEM. Perhaps "assert(r == 0 || errno == ENOMEM)" as both
belt'n'braces and doc?

Actually, pthread_atfork(3) says it "returns an error code" so maybe
that should be "r == 0 || r == ENOMEM" ? Seems oddly inconsistent to
norms though.

The above wouldn't stop me from acking though:

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Ian.

> +
> +    atfork_registered = 1;
> +    rc = 0;
> + out:
> +    atfork_unlock();
> +    return rc;
> +}
> +
> +void libxl__carefd_begin(void) { atfork_lock(); }
> +void libxl__carefd_unlock(void) { atfork_unlock(); }
> +
> +libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
> +{
> +    libxl__carefd *cf = 0;
> +
> +    libxl_fd_set_cloexec(ctx, fd, 1);
> +    cf = libxl__zalloc(NULL, sizeof(*cf));
> +    cf->fd = fd;
> +    LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry);
> +    return cf;
> +}
> +
> +libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd)
> +{
> +    libxl__carefd *cf = 0;
> +
> +    cf = libxl__carefd_record(ctx, fd);
> +    libxl__carefd_unlock();
> +    return cf;
> +}
> +
> +void libxl_postfork_child_noexec(libxl_ctx *ctx)
> +{
> +    libxl__carefd *cf, *cf_tmp;
> +    int r;
> +
> +    atfork_lock();
> +    LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
> +        if (cf->fd >= 0) {
> +            r = close(cf->fd);
> +            if (r)
> +                LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING,
> +                                 "failed to close fd=%d"
> +                                 " (perhaps of another libxl ctx)", cf->fd);
> +        }
> +        free(cf);
> +    }
> +    LIBXL_LIST_INIT(&carefds);
> +    atfork_unlock();
> +}
> +
> +int libxl__carefd_close(libxl__carefd *cf)
> +{
> +    if (!cf) return 0;
> +    int r = cf->fd < 0 ? 0 : close(cf->fd);
> +    int esave = errno;
> +    atfork_lock();
> +    LIBXL_LIST_REMOVE(cf, entry);
> +    atfork_unlock();
> +    free(cf);
> +    errno = esave;
> +    return r;
> +}
> +
> +int libxl__carefd_fd(const libxl__carefd *cf)
> +{
> +    if (!cf) return -1;
> +    return cf->fd;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 740bde2..a8372bb 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -611,6 +611,9 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
>  void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
> 
> 
> +int libxl__atfork_init(libxl_ctx *ctx);
> +
> +
>  /* from xl_dom */
>  _hidden libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid);
>  _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
> @@ -1307,6 +1310,66 @@ _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
>  /* For use by ao machinery ONLY */
>  _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
> 
> +
> +/*
> + * File descriptors and CLOEXEC
> + */
> +
> +/*
> + * For libxl functions which create file descriptors, at least one
> + * of the following must be true:
> + *  (a) libxl does not care if copies of this open-file are inherited
> + *      by random children and might remain open indefinitely
> + *  (b) libxl must take extra care for the fd (the actual descriptor,
> + *      not the open-file) as below.  We call this a "carefd".
> + *
> + * The rules for opening a carefd are:
> + *  (i)   Before bringing any carefds into existence,
> + *        libxl code must call libxl__carefd_begin.
> + *  (ii)  Then for each carefd brought into existence,
> + *        libxl code must call libxl__carefd_record
> + *        and remember the libxl__carefd_record*.
> + *  (iii) Then it must call libxl__carefd_unlock.
> + *  (iv)  When in a child process the fd is to be passed across
> + *        exec by libxl, the libxl code must unset FD_CLOEXEC
> + *        on the fd eg by using libxl_fd_set_cloexec.
> + *  (v)   Later, when the fd is to be closed in the same process,
> + *        libxl code must not call close.  Instead, it must call
> + *        libxl__carefd_close.
> + * Steps (ii) and (iii) can be combined by calling the convenience
> + * function libxl__carefd_opened.
> + */
> +/* libxl__carefd_begin and _unlock (or _opened) must be called always
> + * in pairs.  They may be called with the CTX lock held.  In between
> + * _begin and _unlock, the following are prohibited:
> + *   - anything which might block
> + *   - any callbacks to the application
> + *   - nested calls to libxl__carefd_begin
> + *   - fork (libxl__fork)
> + * In general nothing should be done before _unlock that could be done
> + * afterwards.
> + */
> +typedef struct libxl__carefd libxl__carefd;
> +
> +void libxl__carefd_begin(void);
> +void libxl__carefd_unlock(void);
> +
> +/* fd may be -1, in which case this returns a dummy libxl__fd_record
> + * on which it _carefd_close is a no-op.  Cannot fail. */
> +libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd);
> +
> +/* Combines _record and _unlock in a single call.  If fd==-1,
> + * still does the unlock, but returns 0.  Cannot fail. */
> +libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd);
> +
> +/* Works just like close(2).  You may pass NULL, in which case it's
> + * a successful no-op. */
> +int libxl__carefd_close(libxl__carefd*);
> +
> +/* You may pass NULL in which case the answer is -1. */
> +int libxl__carefd_fd(const libxl__carefd*);
> +
> +
>  /*
>   * Convenience macros.
>   */
> diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
> index 62c0abd..a6ffd25 100644
> --- a/tools/libxl/xl.c
> +++ b/tools/libxl/xl.c
> @@ -96,6 +96,9 @@ static void parse_global_config(const char *configfile,
> 
>  void postfork(void)
>  {
> +    libxl_postfork_child_noexec(ctx); /* in case we don't exit/exec */
> +    ctx = 0;
> +
>      if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
>          fprintf(stderr, "cannot reinit xl context after fork\n");
>          exit(-1);
> --
> 1.7.2.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH 19/31] libxl: provide STATE_AO_GC

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> Provide a convenience macro for use in ao callback functions, and
> document that it should be used.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_internal.h |   11 ++++++++---
>  1 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index a8372bb..76875bb 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1266,9 +1266,10 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
>   * - Note that during callback functions, two gcs are available:
>   *    - The one in egc, whose lifetime is only this callback
>   *    - The one in ao, whose lifetime is the asynchronous operation
> - *   Usually callback function should use CONTAINER_OF
> - *   to obtain its own structure, containing a pointer to the ao,
> - *   and then use the gc from that ao.
> + *   Usually callback function should use CONTAINER_OF to obtain its
> + *   own state structure, containing a pointer to the ao.  It should
> + *   then obtain the ao and use the ao's gc; this is most easily done
> + *   using the convenience macro STATE_AO_GC.
>   */
>  
>  #define AO_CREATE(ctx, domid, ao_how)                           \
> @@ -1298,6 +1299,10 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
>  #define AO_GC                                   \
>      libxl__gc *const gc = &ao->gc
>  
> +#define STATE_AO_GC(op)                         \
> +    libxl__ao *const ao = op->ao;               \
> +    AO_GC
> +
>  
>  /* All of these MUST be called with the ctx locked.
>   * libxl__ao_inprogress MUST be called with the ctx locked exactly once. */

Re: [PATCH 20/31] libxl: handle POLLERR, POLLHUP, POLLNVAL properly

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> Pass POLLERR and POLLHUP to fd callbacks, as is necessary.
> Crash on POLLNVAL since that means our fds are messed up.
> 
> Document the behaviour (including the fact that poll sometimes sets
> POLLHUP or POLLERR even if only POLLIN was requested.
> 
> Fix the one current fd callback to do something with POLLERR|POLLHUP.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> ---
>  tools/libxl/libxl_event.c    |    7 ++++++-
>  tools/libxl/libxl_internal.h |    5 +++++
>  2 files changed, 11 insertions(+), 1 deletions(-)
> 
> diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
> index 638b9ab..5e1a207 100644
> --- a/tools/libxl/libxl_event.c
> +++ b/tools/libxl/libxl_event.c
> @@ -335,6 +335,9 @@ static void watchfd_callback(libxl__egc *egc, libxl__ev_fd *ev,
>  {
>      EGC_GC;
>  
> +    if (revents & (POLLERR|POLLHUP))
> +        LIBXL__EVENT_DISASTER(egc, "unexpected poll event on watch fd", 0, 0);

If this can happen even when we didn't ask for it then is that really a
disaster? Could we just log it and move on (or ignore it)?

Under what circumstances does poll actually behave this way? Is it an
indication that something has gone horribly wrong already?

> +
>      for (;;) {
>          char **event = xs_check_watch(CTX->xsh);
>          if (!event) {
> @@ -739,7 +742,9 @@ static int afterpoll_check_fd(libxl__poller *poller,
>          /* again, stale slot entry */
>          return 0;
>  
> -    int revents = fds[slot].revents & events;
> +    assert(!(fds[slot].revents & POLLNVAL));
> +
> +    int revents = fds[slot].revents & (events | POLLERR | POLLHUP);
>      /* we mask in case requested events have changed */
>  
>      return revents;
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 76875bb..9f2583d 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -127,6 +127,11 @@ void libxl__alloc_failed(libxl_ctx *, const char *func,
>  typedef struct libxl__ev_fd libxl__ev_fd;
>  typedef void libxl__ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
>                                     int fd, short events, short revents);
> +  /* Note that revents may contain POLLERR or POLLHUP regardless of
> +   * events; otherwise revents contains only bits in events.  Contrary
> +   * to the documentation for poll(2), POLLERR and POLLHUP can occur
> +   * even if only POLLIN was set in events.  (POLLNVAL is a fatal
> +   * error and will cause libxl event machinery to fail an assertion.) */
>  struct libxl__ev_fd {
>      /* caller should include this in their own struct */
>      /* read-only for caller, who may read only when registered: */

Re: [PATCH 22/31] libxl: event API: new facilities for waiting for subprocesses

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> The current arrangements in libxl for spawning subprocesses have two
> key problems: (i) they make unwarranted (and largely undocumented)
> assumptions about the caller's use of subprocesses, (ii) they aren't
> integrated into the event system and can't be made asynchronous etc.
> 
> So replace them with a new set of facilities.
> 
> Primarily, from the point of view of code inside libxl, this is
> libxl__ev_child_fork which is both (a) a version of fork() and (b) an
> event source which generates a callback when the child dies.
> 
> From the point of view of the application, we fully document our use
> of SIGCHLD.  The application can tell us whether it wants to own
> SIGCHLD or not; if it does, it has to tell us about deaths of our
> children.
> 
> Currently there are no callers in libxl which use these facilities.
> All code in libxl which forks needs to be converted and libxl_fork
> needse to be be abolished.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Roger Pau Monne <roger.pau@entel.upc.edu>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl.c          |   17 +++-
>  tools/libxl/libxl.h          |    1 +
>  tools/libxl/libxl_event.c    |   53 +++++++--
>  tools/libxl/libxl_event.h    |  147 +++++++++++++++++++++++-
>  tools/libxl/libxl_fork.c     |  255 ++++++++++++++++++++++++++++++++++++++++++
>  tools/libxl/libxl_internal.h |   59 +++++++++-
>  6 files changed, 512 insertions(+), 20 deletions(-)
> 
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 60dbfdc..42ac89f 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -39,7 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
>      memset(ctx, 0, sizeof(libxl_ctx));
>      ctx->lg = lg;
> 
> -    /* First initialise pointers (cannot fail) */
> +    /* First initialise pointers etc. (cannot fail) */
> 
>      LIBXL_TAILQ_INIT(&ctx->occurred);
> 
> @@ -58,6 +58,11 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
>      LIBXL_TAILQ_INIT(&ctx->death_list);
>      libxl__ev_xswatch_init(&ctx->death_watch);
> 
> +    ctx->childproc_hooks = &libxl__childproc_default_hooks;
> +    ctx->childproc_user = 0;
> +
> +    ctx->sigchld_selfpipe[0] = -1;
> +
>      /* The mutex is special because we can't idempotently destroy it */
> 
>      if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
> @@ -160,6 +165,16 @@ int libxl_ctx_free(libxl_ctx *ctx)
> 
>      discard_events(&ctx->occurred);
> 
> +    /* If we have outstanding children, then the application inherits
> +     * them; we wish the application good luck with understanding
> +     * this if and when it reaps them. */
> +    libxl__sigchld_removehandler(ctx);
> +
> +    if (ctx->sigchld_selfpipe[0] >= 0) {
> +        close(ctx->sigchld_selfpipe[0]);
> +        close(ctx->sigchld_selfpipe[1]);
> +    }
> +
>      pthread_mutex_destroy(&ctx->lock);
> 
>      GC_FREE;
> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
> index edbca53..03e71f6 100644
> --- a/tools/libxl/libxl.h
> +++ b/tools/libxl/libxl.h
> @@ -380,6 +380,7 @@ enum {
>      ERROR_NOT_READY = -11,
>      ERROR_OSEVENT_REG_FAIL = -12,
>      ERROR_BUFFERFULL = -13,
> +    ERROR_UNKNOWN_CHILD = -14,
>  };
> 
> 
> diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
> index 672e3fe..1b7495a 100644
> --- a/tools/libxl/libxl_event.c
> +++ b/tools/libxl/libxl_event.c
> @@ -623,6 +623,10 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
>                                                                         \
>          REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
>                                                                         \
> +        int selfpipe = libxl__fork_selfpipe_active(CTX);               \
> +        if (selfpipe >= 0)                                             \
> +            REQUIRE_FD(selfpipe, POLLIN, BODY);                        \
> +                                                                       \
>      }while(0)
> 
>  #define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
> @@ -762,10 +766,11 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
>                                 int nfds, const struct pollfd *fds,
>                                 struct timeval now)
>  {
> +    /* May make callbacks into the application for child processes.
> +     * ctx must be locked exactly once */
>      EGC_GC;
>      libxl__ev_fd *efd;
> 
> -
>      LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
>          if (!efd->events)
>              continue;
> @@ -776,11 +781,16 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
>      }
> 
>      if (afterpoll_check_fd(poller,fds,nfds, poller->wakeup_pipe[0],POLLIN)) {
> -        char buf[256];
> -        int r = read(poller->wakeup_pipe[0], buf, sizeof(buf));
> -        if (r < 0)
> -            if (errno != EINTR && errno != EWOULDBLOCK)
> -                LIBXL__EVENT_DISASTER(egc, "read wakeup", errno, 0);
> +        int e = libxl__self_pipe_eatall(poller->wakeup_pipe[0]);
> +        if (e) LIBXL__EVENT_DISASTER(egc, "read wakeup", e, 0);
> +    }
> +
> +    int selfpipe = libxl__fork_selfpipe_active(CTX);
> +    if (selfpipe >= 0 &&
> +        afterpoll_check_fd(poller,fds,nfds, selfpipe, POLLIN)) {
> +        int e = libxl__self_pipe_eatall(selfpipe);
> +        if (e) LIBXL__EVENT_DISASTER(egc, "read sigchld pipe", e, 0);
> +        libxl__fork_selfpipe_woken(egc);
>      }
> 
>      for (;;) {
> @@ -1078,16 +1088,37 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p)
> 
>  void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p)
>  {
> +    int e = libxl__self_pipe_wakeup(p->wakeup_pipe[1]);
> +    if (e) LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", e, 0);
> +}
> +
> +int libxl__self_pipe_wakeup(int fd)
> +{
>      static const char buf[1] = "";
> 
>      for (;;) {
> -        int r = write(p->wakeup_pipe[1], buf, 1);
> -        if (r==1) return;
> +        int r = write(fd, buf, 1);
> +        if (r==1) return 0;
>          assert(r==-1);
>          if (errno == EINTR) continue;
> -        if (errno == EWOULDBLOCK) return;
> -        LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", errno, 0);
> -        return;
> +        if (errno == EWOULDBLOCK) return 0;
> +        assert(errno);
> +        return errno;
> +    }
> +}
> +
> +int libxl__self_pipe_eatall(int fd)
> +{
> +    char buf[256];
> +    for (;;) {
> +        int r = read(fd, buf, sizeof(buf));
> +        if (r == sizeof(buf)) continue;
> +        if (r >= 0) return 0;
> +        assert(r == -1);
> +        if (errno == EINTR) continue;
> +        if (errno == EWOULDBLOCK) return 0;
> +        assert(errno);
> +        return errno;
>      }
>  }
> 
> diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
> index 2d2196f..713d96d 100644
> --- a/tools/libxl/libxl_event.h
> +++ b/tools/libxl/libxl_event.h
> @@ -163,11 +163,6 @@ void libxl_event_register_callbacks(libxl_ctx *ctx,
>   * After libxl_ctx_free, all corresponding evgen handles become
>   * invalid and must no longer be passed to evdisable.
>   *
> - * Events enabled with evenable prior to a fork and libxl_ctx_postfork
> - * are no longer generated after the fork/postfork; however the evgen
> - * structures are still valid and must be passed to evdisable if the
> - * memory they use should not be leaked.
> - *
>   * Applications should ensure that they eventually retrieve every
>   * event using libxl_event_check or libxl_event_wait, since events
>   * which occur but are not retreived by the application will be queued
> @@ -372,6 +367,148 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
>  void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
> 
> 
> +/*======================================================================*/
> +
> +/*
> + * Subprocess handling.
> + *
> + * Unfortunately the POSIX interface makes this very awkward.
> + *
> + * There are two possible arrangements for collecting statuses from
> + * wait/waitpid.
> + *
> + * For naive programs:
> + *
> + *     libxl will keep a SIGCHLD handler installed whenever it has an
> + *     active (unreaped) child.  It will reap all children with
> + *     wait(); any children it does not recognise will be passed to
> + *     the application via an optional callback (and will result in
> + *     logged warnings if no callback is provided or the callback
> + *     denies responsibility for the child).
> + *
> + *     libxl may have children whenever:
> + *
> + *       - libxl is performing an operation which can be made
> + *         asynchronous; ie one taking a libxl_asyncop_how, even
> + *         if NULL is passed indicating that the operation is
> + *         synchronous; or
> + *
> + *       - events of any kind are being generated, as requested
> + *         by libxl_evenable_....
> + *
> + *     A multithreaded application which is naive in this sense may
> + *     block SIGCHLD on some of its threads, but there must be at
> + *     least one thread that has SIGCHLD unblocked.  libxl will not
> + *     modify the blocking flag for SIGCHLD (except that it may create
> + *     internal service threads with all signals blocked).
> + *
> + *     A naive program must only have at any one time only
> + *     one libxl context which might have children.
> + *
> + * For programs which run their own children alongside libxl's:
> + *
> + *     A program which does this must call libxl_childproc_setmode.
> + *     There are two options:
> + *
> + *     libxl_sigchld_owner_mainloop:
> + *       The application must install a SIGCHLD handler and reap (at
> + *       least) all of libxl's children and pass their exit status
> + *       to libxl by calling libxl_childproc_exited.
> + *
> + *     libxl_sigchld_owner_libxl_always:
> + *       The application expects libxl to reap all of its children,
> + *       and provides a callback to be notified of their exit
> + *       statues.
> + *
> + * An application which fails to call setmode, or which passes 0 for
> + * hooks, while it uses any libxl operation which might
> + * create or use child processes (see above):
> + *   - Must not have any child processes running.
> + *   - Must not install a SIGCHLD handler.
> + *   - Must not reap any children.
> + */
> +
> +
> +typedef enum {
> +    /* libxl owns SIGCHLD whenever it has a child. */
> +    libxl_sigchld_owner_libxl,
> +
> +    /* Application promises to call libxl_childproc_exited but NOT
> +     * from within a signal handler.  libxl will not itself arrange to
> +     * (un)block or catch SIGCHLD. */
> +    libxl_sigchld_owner_mainloop,
> +
> +    /* libxl owns SIGCHLD all the time, and the application is
> +     * relying on libxl's event loop for reaping its own children. */
> +    libxl_sigchld_owner_libxl_always,
> +} libxl_sigchld_owner;
> +
> +typedef struct {
> +    libxl_sigchld_owner chldowner;
> +
> +    /* All of these are optional: */
> +
> +    /* Called by libxl instead of fork.  Should behave exactly like
> +     * fork, including setting errno etc.  May NOT reenter into libxl.
> +     * Application may use this to discover pids of libxl's children,
> +     * for example.
> +     */
> +    pid_t (*fork_replacement)(void *user);
> +
> +    /* With libxl_sigchld_owner_libxl, called by libxl when it has
> +     * reaped a pid.  (Not permitted with _owner_mainloop.)
> +     *
> +     * Should return 0 if the child was recognised by the application
> +     * (or if the application does not keep those kind of records),
> +     * ERROR_UNKNOWN_CHILD if the application knows that the child is not
> +     * the application's; if it returns another error code it is a
> +     * disaster as described for libxl_event_register_callbacks.
> +     * (libxl will report unexpected children to its error log.)
> +     *
> +     * If not supplied, the application is assumed not to start
> +     * any children of its own.
> +     *
> +     * This function is NOT called from within the signal handler.
> +     * Rather it will be called from inside a libxl's event handling
> +     * code and thus only when libxl is running, for example from
> +     * within libxl_event_wait.  (libxl uses the self-pipe trick
> +     * to implement this.)
> +     *
> +     * childproc_exited_callback may call back into libxl, but it
> +     * is best to avoid making long-running libxl calls as that might
> +     * stall the calling event loop while the nested operation
> +     * completes.
> +     */
> +    int (*reaped_callback)(pid_t, int status, void *user);
> +} libxl_childproc_hooks;
> +
> +/* hooks may be 0 in which is equivalent to &{ libxl_sigchld_owner_libxl, 0, 0 }
> + *
> + * May not be called when libxl might have any child processes, or the
> + * behaviour is undefined.  So it is best to call this at
> + * initialisation.
> + */
> +void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
> +                             void *user);
> +
> +/*
> + * This function is for an application which owns SIGCHLD and which
> + * therefore reaps all of the process's children.
> + *
> + * May be called only by an application which has called setmode with
> + * chldowner == libxl_sigchld_owner_mainloop.  If pid was a process started
> + * by this instance of libxl, returns 0 after doing whatever
> + * processing is appropriate.  Otherwise silently returns
> + * ERROR_UNKNOWN_CHILD.  No other error returns are possible.
> + *
> + * May NOT be called from within a signal handler which might
> + * interrupt any libxl operation.  The application will almost
> + * certainly need to use the self-pipe trick (or a working pselect or
> + * ppoll) to implement this.
> + */
> +int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status);
> +
> +
>  /*
>   * An application which initialises a libxl_ctx in a parent process
>   * and then forks a child which does not quickly exec, must
> diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
> index 4751ef4..aac9598 100644
> --- a/tools/libxl/libxl_fork.c
> +++ b/tools/libxl/libxl_fork.c
> @@ -46,6 +46,12 @@ static int atfork_registered;
>  static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
>      LIBXL_LIST_HEAD_INITIALIZER(carefds);
> 
> +/* non-null iff installed, protected by no_forking */
> +static libxl_ctx *sigchld_owner;
> +static struct sigaction sigchld_saved_action;
> +
> +static void sigchld_removehandler_core(void);
> +
>  static void atfork_lock(void)
>  {
>      int r = pthread_mutex_lock(&no_forking);
> @@ -104,6 +110,7 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
>      int r;
> 
>      atfork_lock();
> +
>      LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
>          if (cf->fd >= 0) {
>              r = close(cf->fd);
> @@ -115,6 +122,10 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
>          free(cf);
>      }
>      LIBXL_LIST_INIT(&carefds);
> +
> +    if (sigchld_owner)
> +        sigchld_removehandler_core();
> +
>      atfork_unlock();
>  }
> 
> @@ -138,6 +149,250 @@ int libxl__carefd_fd(const libxl__carefd *cf)
>  }
> 
>  /*
> + * Actual child process handling
> + */
> +
> +static void sigchld_handler(int signo)
> +{
> +    int e = libxl__self_pipe_wakeup(sigchld_owner->sigchld_selfpipe[1]);
> +    assert(!e); /* errors are probably EBADF, very bad */
> +}
> +
> +static void sigchld_removehandler_core(void)
> +{
> +    struct sigaction was;
> +    int r;
> +
> +    r = sigaction(SIGCHLD, &sigchld_saved_action, &was);
> +    assert(!r);
> +    assert(!(was.sa_flags & SA_SIGINFO));
> +    assert(was.sa_handler == sigchld_handler);
> +    sigchld_owner = 0;
> +}
> +
> +void libxl__sigchld_removehandler(libxl_ctx *ctx) /* non-reentrant */
> +{
> +    atfork_lock();
> +    if (sigchld_owner == ctx)
> +        sigchld_removehandler_core();
> +    atfork_unlock();
> +}
> +
> +int libxl__sigchld_installhandler(libxl_ctx *ctx) /* non-reentrant */
> +{
> +    int r, rc;
> +
> +    if (ctx->sigchld_selfpipe[0] < 0) {
> +        r = pipe(ctx->sigchld_selfpipe);
> +        if (r) {
> +            ctx->sigchld_selfpipe[0] = -1;
> +            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
> +                             "failed to create sigchld pipe");
> +            rc = ERROR_FAIL;
> +            goto out;
> +        }
> +    }
> +
> +    atfork_lock();
> +    if (sigchld_owner != ctx) {
> +        struct sigaction ours;
> +
> +        assert(!sigchld_owner);
> +        sigchld_owner = ctx;
> +
> +        memset(&ours,0,sizeof(ours));
> +        ours.sa_handler = sigchld_handler;
> +        sigemptyset(&ours.sa_mask);
> +        ours.sa_flags = SA_NOCLDSTOP | SA_RESTART;
> +        r = sigaction(SIGCHLD, &ours, &sigchld_saved_action);
> +        assert(!r);
> +
> +        assert(((void)"application must negotiate with libxl about SIGCHLD",
> +                !(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
> +                (sigchld_saved_action.sa_handler == SIG_DFL ||
> +                 sigchld_saved_action.sa_handler == SIG_IGN)));
> +    }
> +    atfork_unlock();
> +
> +    rc = 0;
> + out:
> +    return rc;
> +}
> +
> +static int chldmode_ours(libxl_ctx *ctx)
> +{
> +    return ctx->childproc_hooks->chldowner == libxl_sigchld_owner_libxl;
> +}
> +
> +int libxl__fork_selfpipe_active(libxl_ctx *ctx)
> +{
> +    /* Returns the fd to read, or -1 */
> +    if (!chldmode_ours(ctx))
> +        return -1;
> +
> +    if (LIBXL_LIST_EMPTY(&ctx->children))
> +        return -1;
> +
> +    return ctx->sigchld_selfpipe[0];
> +}
> +
> +static void perhaps_removehandler(libxl_ctx *ctx)
> +{
> +    if (LIBXL_LIST_EMPTY(&ctx->children) &&
> +        ctx->childproc_hooks->chldowner != libxl_sigchld_owner_libxl_always)
> +        libxl__sigchld_removehandler(ctx);
> +}
> +
> +static int childproc_reaped(libxl__egc *egc, pid_t pid, int status)
> +{
> +    EGC_GC;
> +    libxl__ev_child *ch;
> +
> +    LIBXL_LIST_FOREACH(ch, &CTX->children, entry)
> +        if (ch->pid == pid)
> +            goto found;
> +
> +    /* not found */
> +    return ERROR_UNKNOWN_CHILD;
> +
> + found:
> +    LIBXL_LIST_REMOVE(ch, entry);
> +    ch->pid = -1;
> +    ch->callback(egc, ch, pid, status);
> +
> +    perhaps_removehandler(CTX);
> +
> +    return 0;
> +}
> +
> +int libxl_childproc_reaped(libxl_ctx *ctx, pid_t pid, int status)
> +{
> +    EGC_INIT(ctx);
> +    CTX_LOCK;
> +    int rc = childproc_reaped(egc, pid, status);
> +    CTX_UNLOCK;
> +    EGC_FREE;
> +    return rc;
> +}
> +
> +void libxl__fork_selfpipe_woken(libxl__egc *egc)
> +{
> +    /* May make callbacks into the application for child processes.
> +     * ctx must be locked EXACTLY ONCE */
> +    EGC_GC;
> +
> +    while (chldmode_ours(CTX) /* in case the app changes the mode */) {
> +        int status;
> +        pid_t pid = waitpid(-1, &status, WNOHANG);
> +
> +        if (pid == 0) return;
> +
> +        if (pid == -1) {
> +            if (errno == ECHILD) return;
> +            if (errno == EINTR) continue;
> +            LIBXL__EVENT_DISASTER(egc, "waitpid() failed", errno, 0);
> +            return;
> +        }
> +
> +        int rc = childproc_reaped(egc, pid, status);
> +
> +        if (rc) {
> +            if (CTX->childproc_hooks->reaped_callback) {
> +                CTX_UNLOCK;
> +                rc = CTX->childproc_hooks->reaped_callback
> +                        (pid, status, CTX->childproc_user);
> +                CTX_LOCK;
> +                if (rc != 0 && rc != ERROR_UNKNOWN_CHILD) {
> +                    char disasterbuf[200];
> +                    snprintf(disasterbuf, sizeof(disasterbuf), " reported by"
> +                             " libxl_childproc_hooks->reaped_callback"
> +                             " (for pid=%lu, status=%d; error code %d)",
> +                             (unsigned long)pid, status, rc);
> +                    LIBXL__EVENT_DISASTER(egc, disasterbuf, 0, 0);
> +                    return;
> +                }
> +            } else {
> +                rc = ERROR_UNKNOWN_CHILD;
> +            }
> +            if (rc)
> +                libxl_report_child_exitstatus(CTX, XTL_WARN,
> +                                "unknown child", (long)pid, status);
> +        }
> +    }
> +}
> +
> +pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *ch,
> +                           libxl__ev_child_callback *death)
> +{
> +    CTX_LOCK;
> +    int rc;
> +
> +    if (chldmode_ours(CTX)) {
> +        rc = libxl__sigchld_installhandler(CTX);
> +        if (rc) goto out;
> +    }
> +
> +    pid_t pid =
> +        CTX->childproc_hooks->fork_replacement
> +        ? CTX->childproc_hooks->fork_replacement(CTX->childproc_user)
> +        : fork();
> +    if (pid == -1) {
> +        LOGE(ERROR, "fork failed");
> +        rc = ERROR_FAIL;
> +        goto out;
> +    }
> +
> +    if (!pid) {
> +        /* woohoo! */
> +        return 0; /* Yes, CTX is left locked in the child. */
> +    }
> +
> +    ch->pid = pid;
> +    ch->callback = death;
> +    LIBXL_LIST_INSERT_HEAD(&CTX->children, ch, entry);
> +    rc = pid;
> +
> + out:
> +    perhaps_removehandler(CTX);
> +    CTX_UNLOCK;
> +    return rc;
> +}
> +
> +void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
> +                             void *user)
> +{
> +    GC_INIT(ctx);
> +    CTX_LOCK;
> +
> +    assert(LIBXL_LIST_EMPTY(&CTX->children));
> +
> +    if (!hooks)
> +        hooks = &libxl__childproc_default_hooks;
> +
> +    ctx->childproc_hooks = hooks;
> +    ctx->childproc_user = user;
> +
> +    switch (ctx->childproc_hooks->chldowner) {
> +    case libxl_sigchld_owner_mainloop:
> +    case libxl_sigchld_owner_libxl:
> +        libxl__sigchld_removehandler(ctx);
> +        break;
> +    case libxl_sigchld_owner_libxl_always:
> +        libxl__sigchld_installhandler(ctx);
> +        break;
> +    default:
> +        abort();
> +    }
> +
> +    CTX_UNLOCK;
> +    GC_FREE;
> +}
> +
> +const libxl_childproc_hooks libxl__childproc_default_hooks = {
> +    libxl_sigchld_owner_libxl, 0, 0
> +};
> +
> +/*
>   * Local variables:
>   * mode: C
>   * c-basic-offset: 4
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 1a2139c..a60171c 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -196,6 +196,19 @@ typedef struct libxl__ev_watch_slot {
>  libxl__ev_xswatch *libxl__watch_slot_contents(libxl__gc *gc, int slotnum);
> 
> 
> +typedef struct libxl__ev_child libxl__ev_child;
> +typedef void libxl__ev_child_callback(libxl__egc *egc, libxl__ev_child*,
> +                                      pid_t pid, int status);
> +struct libxl__ev_child {
> +    /* caller should include this in their own struct */
> +    /* read-only for caller: */
> +    pid_t pid; /* -1 means unused ("unregistered", ie Idle) */
> +    libxl__ev_child_callback *callback;
> +    /* remainder is private for libxl__ev_... */
> +    LIBXL_LIST_ENTRY(struct libxl__ev_child) entry;
> +};
> +
> +
>  /*
>   * evgen structures, which are the state we use for generating
>   * events for the caller.
> @@ -315,10 +328,14 @@ struct libxl__ctx {
> 
>      LIBXL_LIST_HEAD(, libxl_evgen_disk_eject) disk_eject_evgens;
> 
> -    /* for callers who reap children willy-nilly; caller must only
> -     * set this after libxl_init and before any other call - or
> -     * may leave them untouched */
> +    const libxl_childproc_hooks *childproc_hooks;
> +    void *childproc_user;
> +    int sigchld_selfpipe[2]; /* [0]==-1 means handler not installed */
> +    LIBXL_LIST_HEAD(, libxl__ev_child) children;
> +
> +    /* This is obsolete and must be removed: */
>      int (*waitpid_instead)(pid_t pid, int *status, int flags);
> +
>      libxl_version_info version_info;
>  };
> 
> @@ -566,6 +583,33 @@ static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw)
> 
> 
>  /*
> + * For making subprocesses.  This is the only permitted mechanism for
> + * code in libxl to do so.
> + *
> + * In the parent, returns the pid, filling in childw_out.
> + * In the child, returns 0.
> + * If it fails, returns a libxl error (all of which are -ve).
> + *
> + * The child should go on to exec (or exit) soon, and should not make
> + * any further libxl event calls in the meantime.
> + *
> + * The parent may signal the child but it must not reap it.  That will
> + * be done by the event machinery.  death may be NULL in which case
> + * the child is still reaped but its death is ignored.
> + *
> + * It is not possible to "deregister" the child death event source.
> + * It will generate exactly one event callback; until then the childw
> + * is Active and may not be reused.
> + */
> +_hidden pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *childw_out,
> +                                 libxl__ev_child_callback *death);
> +static inline void libxl__ev_child_init(libxl__ev_child *childw_out)
> +                { childw_out->pid = -1; }
> +static inline int libxl__ev_child_inuse(libxl__ev_child *childw_out)
> +                { return childw_out->pid >= 0; }
> +
> +
> +/*
>   * Other event-handling support provided by the libxl event core to
>   * the rest of libxl.
>   */
> @@ -619,6 +663,15 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
>   * ctx must be locked. */
>  void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
> 
> +/* Internal to fork and child reaping machinery */
> +extern const libxl_childproc_hooks libxl__childproc_default_hooks;
> +int libxl__sigchld_installhandler(libxl_ctx *ctx); /* non-reentrant;logs errs */
> +void libxl__sigchld_removehandler(libxl_ctx *ctx); /* non-reentrant */
> +int libxl__fork_selfpipe_active(libxl_ctx *ctx); /* returns read fd or -1 */
> +void libxl__fork_selfpipe_woken(libxl__egc *egc);
> +int libxl__self_pipe_wakeup(int fd); /* returns 0 or -1 setting errno */
> +int libxl__self_pipe_eatall(int fd); /* returns 0 or -1 setting errno */
> +
> 
>  int libxl__atfork_init(libxl_ctx *ctx);
> 
> --
> 1.7.2.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH 24/31] libxl: provide libxl__remove_file et al.

[Ian Campbell]

On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> These utility functions cope with EINTR AND ENOENT, do error logging,
> and we provide a recursive version to delete whole directory trees.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> ---
>  tools/libxl/libxl_internal.h |    7 ++++
>  tools/libxl/libxl_utils.c    |   79 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 86 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index a60171c..8e47213 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -442,6 +442,13 @@ _hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n);
>   * string. (similar to a gc'd dirname(3)). */
>  _hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s);
>  
> +/* Each of these logs errors and returns a libxl error code.
> + * They do not mind if path is already removed.
> + * For _file, path must not be a directory; for _directory it must be. */
> +_hidden int libxl__remove_file(libxl__gc *gc, const char *path);
> +_hidden int libxl__remove_directory(libxl__gc *gc, const char *path);
> +_hidden int libxl__remove_file_or_directory(libxl__gc *gc, const char *path);
> +
>  _hidden char **libxl__xs_kvs_of_flexarray(libxl__gc *gc, flexarray_t *array, int length);
>  
>  _hidden int libxl__xs_writev(libxl__gc *gc, xs_transaction_t t,
> diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
> index 0cbd85e..c9157a6 100644
> --- a/tools/libxl/libxl_utils.c
> +++ b/tools/libxl/libxl_utils.c
> @@ -364,6 +364,85 @@ int libxl_read_file_contents(libxl_ctx *ctx, const char *filename,
>  READ_WRITE_EXACTLY(read, 1, /* */)
>  READ_WRITE_EXACTLY(write, 0, const)
>  
> +int libxl__remove_file(libxl__gc *gc, const char *path)
> +{
> +    for (;;) {
> +        int r = unlink(path);
> +        if (!r) return 0;
> +        if (errno == ENOENT) return 0;
> +        if (errno == EINTR) continue;
> +        LOGE(ERROR, "failed to remove file %s", path);
> +        return ERROR_FAIL;
> +     }
> +}
> +
> +int libxl__remove_file_or_directory(libxl__gc *gc, const char *path)
> +{
> +    for (;;) {
> +        int r = rmdir(path);
> +        if (!r) return 0;
> +        if (errno == ENOENT) return 0;
> +        if (errno == ENOTEMPTY) return libxl__remove_directory(gc, path);
> +        if (errno == ENOTDIR) return libxl__remove_file(gc, path);
> +        if (errno == EINTR) continue;

starting to look a bit like a switch statement...

> +        LOGE(ERROR, "failed to remove %s", path);
> +        return ERROR_FAIL;
> +     }
> +}
> +
> +int libxl__remove_directory(libxl__gc *gc, const char *dirpath)
> +{
> +    int rc = 0;
> +    DIR *d = 0;
> +
> +    d = opendir(dirpath);
> +    if (!d) {
> +        if (errno == ENOENT)
> +            goto out;
> +
> +        LOGE(ERROR, "failed to opendir %s for removal", dirpath);
> +        rc = ERROR_FAIL;
> +        goto out;
> +    }
> +
> +    size_t need = offsetof(struct dirent, d_name) +
> +        pathconf(dirpath, _PC_NAME_MAX) + 1;
> +    struct dirent *de_buf = libxl__zalloc(gc, need);
> +    struct dirent *de;
> +
> +    for (;;) {
> +        int r = readdir_r(d, de_buf, &de);
> +        if (r) {
> +            LOGE(ERROR, "failed to readdir %s for removal", dirpath);
> +            rc = ERROR_FAIL;
> +            break;
> +        }
> +        if (!de)
> +            break;
> +
> +        if (!strcmp(de->d_name, ".") ||
> +            !strcmp(de->d_name, ".."))
> +            continue;
> +
> +        const char *subpath = GCSPRINTF("%s/%s", dirpath, de->d_name);
> +        if (libxl__remove_file_or_directory(gc, subpath))
> +            rc = ERROR_FAIL;
> +    }
> +
> +    for (;;) {
> +        int r = rmdir(dirpath);
> +        if (!r) break;
> +        if (errno == ENOENT) return 0;

This misses the closedir() at out?

> +        if (errno == EINTR) continue;
> +        LOGE(ERROR, "failed to remove emptied directory %s", dirpath);
> +        rc = ERROR_FAIL;
> +    }
> +
> + out:
> +    if (d) closedir(d);
> +
> +    return rc;
> +}
>  
>  pid_t libxl_fork(libxl_ctx *ctx)
>  {

Re: [PATCH 26/31] libxl: Clean up setdefault in do_domain_create

[Ian Campbell]

On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> do_domain_create called libxl__domain_create_info_setdefault twice.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <Ian.Campbell@citrix.com>

> ---
>  tools/libxl/libxl_create.c |    3 ---
>  1 files changed, 0 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index e63c7bd..3675afe 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -552,9 +552,6 @@ static int do_domain_create(libxl__gc *gc, libxl_domain_config *d_config,
>      ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
>      if (ret) goto error_out;
>  
> -    ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
> -    if (ret) goto error_out;
> -
>      ret = libxl__domain_make(gc, &d_config->c_info, &domid);
>      if (ret) {
>          LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "cannot make domain: %d", ret);

Re: [PATCH 30/31] libxl: make libxl_create_logfile const-correct

[Ian Campbell]

On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_utils.c |    2 +-
>  tools/libxl/libxl_utils.h |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
> index dc25ab0..8eacebf 100644
> --- a/tools/libxl/libxl_utils.c
> +++ b/tools/libxl/libxl_utils.c
> @@ -193,7 +193,7 @@ static int logrename(libxl__gc *gc, const char *old, const char *new)
>      return 0;
>  }
>  
> -int libxl_create_logfile(libxl_ctx *ctx, char *name, char **full_name)
> +int libxl_create_logfile(libxl_ctx *ctx, const char *name, char **full_name)
>  {
>      GC_INIT(ctx);
>      struct stat stat_buf;
> diff --git a/tools/libxl/libxl_utils.h b/tools/libxl/libxl_utils.h
> index ca53a8a..2b47622 100644
> --- a/tools/libxl/libxl_utils.h
> +++ b/tools/libxl/libxl_utils.h
> @@ -26,7 +26,7 @@ int libxl_name_to_cpupoolid(libxl_ctx *ctx, const char *name, uint32_t *poolid);
>  char *libxl_cpupoolid_to_name(libxl_ctx *ctx, uint32_t poolid);
>  int libxl_get_stubdom_id(libxl_ctx *ctx, int guest_domid);
>  int libxl_is_stubdom(libxl_ctx *ctx, uint32_t domid, uint32_t *target_domid);
> -int libxl_create_logfile(libxl_ctx *ctx, char *name, char **full_name);
> +int libxl_create_logfile(libxl_ctx *ctx, const char *name, char **full_name);
>  int libxl_string_to_backend(libxl_ctx *ctx, char *s, libxl_disk_backend *backend);
>  
>  int libxl_read_file_contents(libxl_ctx *ctx, const char *filename,

Re: [PATCH 31/31] libxl: log bootloader output

[Ian Campbell]

On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> This involves adding a new log feature to libxl__datacopier, and then
> using it.
> 
> If the bootloader exits nonzero we print the log filename in a log
> message from libxl.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

This has been an annoying omission for ages, thanks!

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_aoutils.c    |   10 ++++++++++
>  tools/libxl/libxl_bootloader.c |   24 ++++++++++++++++++++++++
>  tools/libxl/libxl_internal.h   |    3 ++-
>  3 files changed, 36 insertions(+), 1 deletions(-)
> 
> diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
> index b33abe4..1b17c84 100644
> --- a/tools/libxl/libxl_aoutils.c
> +++ b/tools/libxl/libxl_aoutils.c
> @@ -118,6 +118,16 @@ static void datacopier_readable(libxl__egc *egc, libxl__ev_fd *ev,
>              libxl__ev_fd_deregister(gc, &dc->toread);
>              break;
>          }
> +        if (dc->log) {
> +            int wrote = fwrite(buf->buf + buf->used, 1, r, dc->log);
> +            if (wrote != r) {
> +                assert(ferror(dc->log));
> +                assert(errno);
> +                LOGE(ERROR, "error logging %s", dc->copywhat);
> +                datacopier_callback(egc, dc, 0, errno);
> +                return;
> +            }
> +        }
>          buf->used += r;
>          dc->used += r;
>          assert(buf->used <= sizeof(buf->buf));
> diff --git a/tools/libxl/libxl_bootloader.c b/tools/libxl/libxl_bootloader.c
> index 1361117..47ced8a 100644
> --- a/tools/libxl/libxl_bootloader.c
> +++ b/tools/libxl/libxl_bootloader.c
> @@ -234,6 +234,10 @@ static void bootloader_cleanup(libxl__egc *egc, libxl__bootloader_state *bl)
>          libxl__carefd_close(bl->ptys[i].master);
>          libxl__carefd_close(bl->ptys[i].slave);
>      }
> +    if (bl->display.log) {
> +        fclose(bl->display.log);
> +        bl->display.log = NULL;
> +    }
>  }
>  
>  static void bootloader_setpaths(libxl__gc *gc, libxl__bootloader_state *bl)
> @@ -256,6 +260,8 @@ void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
>  {
>      STATE_AO_GC(bl);
>      libxl_domain_build_info *info = bl->info;
> +    uint32_t domid = bl->domid;
> +    char *logfile_tmp = NULL;
>      int rc, r;
>  
>      libxl__bootloader_init(bl);
> @@ -268,6 +274,22 @@ void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
>  
>      bootloader_setpaths(gc, bl);
>  
> +    const char *logfile_leaf = GCSPRINTF("bootloader.%"PRIu32, domid);
> +    rc = libxl_create_logfile(CTX, logfile_leaf, &logfile_tmp);
> +    if (rc) goto out;
> +
> +    /* Transfer ownership of log filename to bl and the gc */
> +    bl->logfile = logfile_tmp;
> +    libxl__ptr_add(gc, logfile_tmp);
> +    logfile_tmp = NULL;
> +
> +    bl->display.log = fopen(bl->logfile, "a");
> +    if (!bl->display.log) {
> +        LOGE(ERROR, "failed to create bootloader logfile %s", bl->logfile);
> +        rc = ERROR_FAIL;
> +        goto out;
> +    }
> +
>      for (;;) {
>          r = mkdir(bl->outputdir, 0600);
>          if (!r) break;
> @@ -305,6 +327,7 @@ void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
>      return;
>  
>   out:
> +    free(logfile_tmp);
>      assert(rc);
>      bootloader_callback(egc, bl, rc);
>  }
> @@ -462,6 +485,7 @@ static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
>      libxl__datacopier_kill(&bl->display);
>  
>      if (status) {
> +        LOG(ERROR, "bootloader failed - consult logfile %s", bl->logfile);
>          libxl_report_child_exitstatus(CTX, XTL_ERROR, "bootloader",
>                                        pid, status);
>          rc = ERROR_FAIL;
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index fd96b31..c846144 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1485,6 +1485,7 @@ struct libxl__datacopier_state {
>      int readfd, writefd;
>      ssize_t maxsz;
>      const char *copywhat, *readwhat, *writewhat; /* for error msgs */
> +    FILE *log; /* gets a copy of everything */
>      libxl__datacopier_callback *callback;
>      /* remaining fields are private to datacopier */
>      libxl__ev_fd toread, towrite;
> @@ -1547,7 +1548,7 @@ struct libxl__bootloader_state {
>      libxl_device_disk *disk;
>      uint32_t domid;
>      /* private to libxl__run_bootloader */
> -    char *outputpath, *outputdir;
> +    char *outputpath, *outputdir, *logfile;
>      char *diskpath; /* not from gc, represents actually attached disk */
>      libxl__openpty_state openpty;
>      libxl__openpty_result ptys[2];  /* [0] is for bootloader */

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure"):
> On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
...
> > Consequently,
> >  - New noreturn function libxl__alloc_failed which may be used for
> >    printing a vaguely-useful error message, rather than simply
> >    dereferencing a null pointer.
> 
> We got that in the next patch?

Uh ?  libxl__alloc_failed is in this patch.  I'm not sure what you
mean.

> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Ta.

> > +    int new_maxsize = gc->alloc_maxsize * 2 + 25;
> 
> + 25? (I don't mind, seems even more arbitrary now that we have the *2
> though).

Well, zero isn't adequate :-).  So yes, it's arbitrary.  25 is 100
bytes (i386) or 200 bytes (amd64) which seems a reasonable initial
overhead and will probably avoid triggering a realloc too often.

> > +    assert(new_maxsize < INT_MAX / sizeof(void*) / 2);
> > +    gc->alloc_ptrs = realloc(gc->alloc_ptrs, new_maxsize * sizeof(void *));
> > +    if (!gc->alloc_ptrs)
> > +        libxl__alloc_failed(CTX, __func__, sizeof(void*), new_maxsize);
> 
> Strictly this should be "..., new_maxsize, sizeof(void*)" since the
> arguments are nmemb and size?

I was going to say "we should do this the same way as fwrite and
calloc" so I looked them up, and they have the nmemb and size
arguments in THE OPPOSITE ORDER.  No wonder I can never remember!

I guess this is more like calloc and it should mirror libxl_calloc so
the prototype is right and this call site is wrong.  Fixed.

Ian.

Re: [PATCH v4 00/31] libxl child process handling

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH v4 00/31] libxl child process handling"):
> On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> > Bugfixes for problems reported by Roger Pau Monne:
> >   02/31 libxl: ao: allow immediate completion
> >   03/31 libxl: fix hang due to libxl__initiate_device_remove
> >   04/31 libxl: Fix eventloop_iteration over-locking
> > * 05/31 libxl: remove poller from list in libxl__poller_get
> > 
> > Other general bugfixes:
> > * 01/31 .gitignore: Add a missing file
> >   06/31 libxl: Fix leak of ctx->lock
> >   07/31 tools: Correct PTHREAD options in config/StdGNU.mk
> >   08/31 libxl: Use PTHREAD_CFLAGS, LDFLAGS, LIBS
> >   09/31 tools: Use PTHREAD_CFLAGS, _LDFLAGS, _LIBS
> 
> I think all of the above are now acked and since they are largely
> unrelated to the overall thrust of the series could easily go in now?

Yes.

> >   26/31 libxl: Clean up setdefault in do_domain_create
> >   30/31 libxl: make libxl_create_logfile const-correct
> > 
> > Clarifications and improvements related to memory allocation:
> >   10/31 libxl: Crash (more sensibly) on malloc failure
> >   11/31 libxl: Make libxl__zalloc et al tolerate a NULL gc
> 
> These two could probably go in now too, especially 10/31 would be useful
> to have sooner rather than later so we can stop writing error handling
> code...

Yes.

> Actually the more I read the more I see that a good chunk of the front
> of this series could go in now. Is that what you are planning to do?

Indeed.  I was holding off before because I hadn't actually executed
any of it, but I have now.

I intend to apply as much as I quickly get acks for.

Thanks,
Ian.

Re: [PATCH 18/31] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 18/31] libxl: Protect fds with CLOEXEC even with forking threads"):
> On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> > +    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
> > +    if (r) libxl__alloc_failed(ctx, __func__, 0,0);
> 
> This is a bit subtle -- the only documented error from pthread_atfork is
> ENOMEM. Perhaps "assert(r == 0 || errno == ENOMEM)" as both
> belt'n'braces and doc?

Sure.  I have done this:

      r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
      if (r) {
          assert(r == ENOMEM);
          libxl__alloc_failed(ctx, __func__, 0,0);
      }

> The above wouldn't stop me from acking though:
> 
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks,
Ian.

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Campbell]

On Wed, 2012-04-11 at 11:24 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure"):
> > On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> ...
> > > Consequently,
> > >  - New noreturn function libxl__alloc_failed which may be used for
> > >    printing a vaguely-useful error message, rather than simply
> > >    dereferencing a null pointer.
> > 
> > We got that in the next patch?
> 
> Uh ?  libxl__alloc_failed is in this patch.  I'm not sure what you
> mean.

Quoted the wrong bullet! From the wrong list!

I was trying to respond to:

> Things left to do:
>  - Provide versions of malloc, realloc and free which do the
>    checking but which do not enroll the pointer in the gc.

> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
> 
> Ta.
> 
> > > +    int new_maxsize = gc->alloc_maxsize * 2 + 25;
> > 
> > + 25? (I don't mind, seems even more arbitrary now that we have the *2
> > though).
> 
> Well, zero isn't adequate :-).  So yes, it's arbitrary.  25 is 100
> bytes (i386) or 200 bytes (amd64) which seems a reasonable initial
> overhead and will probably avoid triggering a realloc too often.

Why isn't just doubling each time adequate?

> > > +    assert(new_maxsize < INT_MAX / sizeof(void*) / 2);
> > > +    gc->alloc_ptrs = realloc(gc->alloc_ptrs, new_maxsize * sizeof(void *));
> > > +    if (!gc->alloc_ptrs)
> > > +        libxl__alloc_failed(CTX, __func__, sizeof(void*), new_maxsize);
> > 
> > Strictly this should be "..., new_maxsize, sizeof(void*)" since the
> > arguments are nmemb and size?
> 
> I was going to say "we should do this the same way as fwrite and
> calloc" so I looked them up, and they have the nmemb and size
> arguments in THE OPPOSITE ORDER.  No wonder I can never remember!

I'd never noticed that, how confusing.

> I guess this is more like calloc and it should mirror libxl_calloc so
> the prototype is right and this call site is wrong.  Fixed.
> 
> Ian.

Re: [PATCH 27/31] libxl: provide libxl__datacopier_*

[Ian Campbell]

On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> General facility for ao operations to shovel data between fds.
> 
> This will be used by the bootloader machinery.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
[...]
> +static void datacopier_writable(libxl__egc *egc, libxl__ev_fd *ev,
> +                                int fd, short events, short revents);
> +
> +static void datacopier_check_state(libxl__egc *egc, libxl__datacopier_state *dc)
> +{
> +    STATE_AO_GC(dc);
> +    int rc;
> +
> +    if (dc->used) {
> +        if (!libxl__ev_fd_isregistered(&dc->towrite)) {
> +            rc = libxl__ev_fd_register(gc, &dc->towrite, datacopier_writable,
> +                                       dc->writefd, POLLOUT);
> +            if (rc) {
> +                LOG(ERROR, "unable to establish write event on %s"
> +                    " during copy of %s", dc->writewhat, dc->copywhat);
> +                datacopier_callback(egc, dc, -1, 0);
> +                return;
> +            }
> +        }
> +    } else if (!libxl__ev_fd_isregistered(&dc->toread)) {
> +        /* we have had eof */
> +        libxl__datacopier_kill(dc);
> +        dc->callback(egc, dc, 0, 0);

This looks like an open coded datacopier_callback().

[...]
> @@ -1455,6 +1456,45 @@ int libxl__carefd_close(libxl__carefd*);
>  int libxl__carefd_fd(const libxl__carefd*);
> 
> 
> +/*----- datacopier: copies data from one fd to another -----*/
> +
> +typedef struct libxl__datacopier_state libxl__datacopier_state;
> +typedef struct libxl__datacopier_buf libxl__datacopier_buf;
> +
> +/* onwrite==1 means failure happened when writing, logged, errno is valid
> + * onwrite==0 means failure happened when reading
> + *     errno==0 means we got eof and all data was written
> + *     errno!=0 means we had a read error, logged

ITYM errnoval in every case here rather than actual errno ?

> + * onwrite==-1 means some other internal failure, errnoval not valid, logged
> + * in all cases copier is killed before calling this callback */
> +typedef void libxl__datacopier_callback(libxl__egc *egc,
> +     libxl__datacopier_state *dc, int onwrite, int errnoval);
> +

Ian.

Re: [PATCH 20/31] libxl: handle POLLERR, POLLHUP, POLLNVAL properly

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 20/31] libxl: handle POLLERR, POLLHUP, POLLNVAL properly"):
> On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> > +    if (revents & (POLLERR|POLLHUP))
> > +        LIBXL__EVENT_DISASTER(egc, "unexpected poll event on watch fd", 0, 0);
> 
> If this can happen even when we didn't ask for it then is that really a
> disaster? Could we just log it and move on (or ignore it)?

You never ask for POLLERR or POLLHUP.  They happen regardless.  If
they happen, poll() will keep reporting them.  So we have to stop
polling on this fd entirely - ie effectively the fd has become broken,
unless we know what the cause is and can do something sane about it.
Since this is supposedly the xenstore fd, it is indeed a disaster.

For comparison, if we get eof on the xenstore fd, libxenstore causes
xs_read_watch to return 0 setting errno to EBADF (!) (xs.c:365), which
is also a DISASTER (libxl_event.c:343).

> Under what circumstances does poll actually behave this way? Is it an
> indication that something has gone horribly wrong already?

Yes, exactly.

POLLPRI occurs on sockets and means that the socket has urgent data
(TCP) or out of band data (some other protocols); an application using
TCP would take this as a signal to read data from the fd until it
catches up with the urgent pointer.

POLLHUP occurs when (eg) a pty's other end has been closed and the pty
is shut down.  An application which sees POLLHUP would typically close
the fd.  Eg, in an earlier version of the bootloader rework I had a
bug which would fail to keep an fd open onto the slave side of the
xenconsole pty, so libxl would get POLLHUP on the master side the
bootloader execution would be aborted.  This was handy for my error
testing :-) and is an example of a POLLHUP which isn't a DISASTER,
although it is pretty fatal for the fd in question.

These events should not occur on the xenstore handle.

Ian.

Re: [PATCH 20/31] libxl: handle POLLERR, POLLHUP, POLLNVAL properly

[Ian Campbell]

On Wed, 2012-04-11 at 11:50 +0100, Ian Jackson wrote:
[...snip...]
> These events should not occur on the xenstore handle.

All made sense, thanks for the explanation.

Ian

Re: [PATCH 24/31] libxl: provide libxl__remove_file et al.

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 24/31] libxl: provide libxl__remove_file et al."):
> On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> > +int libxl__remove_file_or_directory(libxl__gc *gc, const char *path)
> > +{
> > +    for (;;) {
> > +        int r = rmdir(path);
> > +        if (!r) return 0;
> > +        if (errno == ENOENT) return 0;
> > +        if (errno == ENOTEMPTY) return libxl__remove_directory(gc, path);
> > +        if (errno == ENOTDIR) return libxl__remove_file(gc, path);
> > +        if (errno == EINTR) continue;
> 
> starting to look a bit like a switch statement...

Perhaps.  I'm not sure I want to obscure the similarity with (and the
differences from) the other analogous error-handling code, and those
other sites typically have only a couple of errno cases making a
switch overkill.  Also of course a switch would switch only on errno
whereas the branching is on r too, so it would add another level of
nesting.

> > +    for (;;) {
> > +        int r = rmdir(dirpath);
> > +        if (!r) break;
> > +        if (errno == ENOENT) return 0;
> 
> This misses the closedir() at out?

Well spotted, fixed.

Ian.

Re: [PATCH 24/31] libxl: provide libxl__remove_file et al.

[Ian Campbell]

On Wed, 2012-04-11 at 11:56 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 24/31] libxl: provide libxl__remove_file et al."):
> > On Tue, 2012-04-10 at 20:07 +0100, Ian Jackson wrote:
> > > +int libxl__remove_file_or_directory(libxl__gc *gc, const char *path)
> > > +{
> > > +    for (;;) {
> > > +        int r = rmdir(path);
> > > +        if (!r) return 0;
> > > +        if (errno == ENOENT) return 0;
> > > +        if (errno == ENOTEMPTY) return libxl__remove_directory(gc, path);
> > > +        if (errno == ENOTDIR) return libxl__remove_file(gc, path);
> > > +        if (errno == EINTR) continue;
> > 
> > starting to look a bit like a switch statement...
> 
> Perhaps.  I'm not sure I want to obscure the similarity with (and the
> differences from) the other analogous error-handling code, and those
> other sites typically have only a couple of errno cases making a
> switch overkill.  Also of course a switch would switch only on errno
> whereas the branching is on r too, so it would add another level of
> nesting.

Sounds reasonable to leave it as is then.

> 
> > > +    for (;;) {
> > > +        int r = rmdir(dirpath);
> > > +        if (!r) break;
> > > +        if (errno == ENOENT) return 0;
> > 
> > This misses the closedir() at out?
> 
> Well spotted, fixed.
> 
> Ian.

Re: [PATCH 28/31] libxl: provide libxl__openpty_*

[Ian Campbell]

On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> General facility for ao operations to open ptys.
> 
> This will be used by the bootloader machinery.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> ---
>  tools/libxl/libxl_aoutils.c  |  127 ++++++++++++++++++++++++++++++++++++++++++
>  tools/libxl/libxl_internal.h |   36 ++++++++++++
>  2 files changed, 163 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
> index c1229e4..b33abe4 100644
> --- a/tools/libxl/libxl_aoutils.c
> +++ b/tools/libxl/libxl_aoutils.c
[...]
> +int libxl__openptys(libxl__openpty_state *op,
> +                    const struct termios *termp,
> +                    const struct winsize *winp) {
> +    /*
> +     * This is completely crazy.

;-)

>   openpty calls grantpt which the spec
> +     * says may fork, and may not be called with a SIGCHLD handler.
> +     * Now our application may have a SIGCHLD handler so that's bad.
> +     * We could perhaps block it but we'd need to block it on all
> +     * threads.  This is just Too Hard.
> +     *
> +     * So instead, we run openpty in a child process.  That child
> +     * process then of course has only our own thread and our own
> +     * signal handlers.  We pass the fds back.
> +     *
> +     * Since our only current caller actually wants two ptys, we
> +     * support calling openpty multiple times for a single fork.
> +     */
> +    STATE_AO_GC(op);
> +    int count = op->count;
> +    int r, i, rc, sockets[2], ptyfds[count][2];
> +    libxl__carefd *for_child = 0;
> +    pid_t pid = -1;
> +
> +    for (i=0; i<count; i++) {
> +        ptyfds[i][0] = ptyfds[i][1] = -1;
> +        libxl__openpty_result *res = &op->results[i];
> +        assert(!res->master);
> +        assert(!res->slave);
> +    }
> +    sockets[0] = sockets[1] = -1; /* 0 is for us, 1 for our child */
> +
> +    libxl__carefd_begin();
> +    r = socketpair(AF_UNIX, SOCK_STREAM, 0, sockets);
> +    if (r) { sockets[0] = sockets[1] = -1; }
> +    for_child = libxl__carefd_opened(CTX, sockets[1]);
> +    if (r) { LOGE(ERROR,"socketpair failed"); rc = ERROR_FAIL; goto out; }
> +
> +    pid = libxl__ev_child_fork(gc, &op->child, openpty_exited);
> +    if (pid == -1) {
> +        rc = ERROR_FAIL;
> +        goto out;
> +    }
> +
> +    if (!pid) {
> +        /* child */
> +        close(sockets[0]);
> +        signal(SIGCHLD, SIG_DFL);
> +
> +        for (i=0; i<count; i++) {
> +            r = openpty(&ptyfds[i][0], &ptyfds[i][1], NULL, termp, winp);
> +            if (r) { LOGE(ERROR,"openpty failed"); _exit(-1); }
> +        }
> +        rc = libxl__sendmsg_fds(gc, sockets[1], "",1,
> +                                2*count, &ptyfds[0][0], "ptys");
> +        if (rc) { LOGE(ERROR,"sendmsg to parent failed"); _exit(-1); }

Buried in this LOGE is a CTX somewhere, right? Is it valid to keep using
it? Perhaps it's OK just for this logging.

I suppose we don't need to call the postfork-noexecing handler because
we are inside libxl?

Ian.

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure"):
> On Wed, 2012-04-11 at 11:24 +0100, Ian Jackson wrote:
> > Things left to do:
> >  - Provide versions of malloc, realloc and free which do the
> >    checking but which do not enroll the pointer in the gc.
...
> We got that in the next patch?

Yes.  I don't think this bullet point is actually helpful in that
commit message as it's not so much a deficiency in the code at that
point as a thing I was intending to fix when I wrote it, so I have
removed it.

> > Well, zero isn't adequate :-).  So yes, it's arbitrary.  25 is 100
> > bytes (i386) or 200 bytes (amd64) which seems a reasonable initial
> > overhead and will probably avoid triggering a realloc too often.
> 
> Why isn't just doubling each time adequate?

alloc_maxsize is initialised to 0.  Doubling zero gives zero.

NB that libxl__ptr_add needs to be rewritten not to be quadratic in
the number of pointrs added (!)

Ian.

Re: [PATCH 27/31] libxl: provide libxl__datacopier_*

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 27/31] libxl: provide libxl__datacopier_*"):
> On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> > +        /* we have had eof */
> > +        libxl__datacopier_kill(dc);
> > +        dc->callback(egc, dc, 0, 0);
> 
> This looks like an open coded datacopier_callback().

Fixed.

> > +/* onwrite==1 means failure happened when writing, logged, errno is valid
> > + * onwrite==0 means failure happened when reading
> > + *     errno==0 means we got eof and all data was written
> > + *     errno!=0 means we had a read error, logged
> 
> ITYM errnoval in every case here rather than actual errno ?

Yes.

Thanks,
Ian.

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Campbell]

On Wed, 2012-04-11 at 12:04 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure"):
> > On Wed, 2012-04-11 at 11:24 +0100, Ian Jackson wrote:
> > > Things left to do:
> > >  - Provide versions of malloc, realloc and free which do the
> > >    checking but which do not enroll the pointer in the gc.
> ...
> > We got that in the next patch?
> 
> Yes.  I don't think this bullet point is actually helpful in that
> commit message as it's not so much a deficiency in the code at that
> point as a thing I was intending to fix when I wrote it, so I have
> removed it.
> 
> > > Well, zero isn't adequate :-).  So yes, it's arbitrary.  25 is 100
> > > bytes (i386) or 200 bytes (amd64) which seems a reasonable initial
> > > overhead and will probably avoid triggering a realloc too often.
> > 
> > Why isn't just doubling each time adequate?
> 
> alloc_maxsize is initialised to 0.  Doubling zero gives zero.

Oh, for some reason I expected it would be initialised to >=1.

> NB that libxl__ptr_add needs to be rewritten not to be quadratic in
> the number of pointrs added (!)

Isn't it O(N) in numbers of pointers?

I also just noticed that the initial:
   /* fast case: we have space in the array for storing the pointer */
    for (i = 0; i < gc->alloc_maxsize; i++) {
        if (!gc->alloc_ptrs[i]) {
            gc->alloc_ptrs[i] = ptr;
            return 0;
        }
    }
is a bit suboptimal since we never remove a ptr from the array, so we
may as well track the max actually used. Then the fast path might
actually be fast...

Ian.

Re: [PATCH 28/31] libxl: provide libxl__openpty_*

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 28/31] libxl: provide libxl__openpty_*"):
> On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> > General facility for ao operations to open ptys.
...
> > +    if (!pid) {
> > +        /* child */
> > +        close(sockets[0]);
> > +        signal(SIGCHLD, SIG_DFL);
> > +
> > +        for (i=0; i<count; i++) {
> > +            r = openpty(&ptyfds[i][0], &ptyfds[i][1], NULL, termp, winp);
> > +            if (r) { LOGE(ERROR,"openpty failed"); _exit(-1); }
> > +        }
> > +        rc = libxl__sendmsg_fds(gc, sockets[1], "",1,
> > +                                2*count, &ptyfds[0][0], "ptys");
> > +        if (rc) { LOGE(ERROR,"sendmsg to parent failed"); _exit(-1); }
> 
> Buried in this LOGE is a CTX somewhere, right? Is it valid to keep using
> it? Perhaps it's OK just for this logging.

This should be documented somewhere.  But I see it isn't.  The closest
we have is:

  * The child should go on to exec (or exit) soon, and should not make
  * any further libxl event calls in the meantime.

I have clarified this:

 * The child should go on to exec (or exit) soon.  The child may not
 * make any further calls to libxl infrastructure, except for memory
 * allocation and logging.  If the child needs to use xenstore it
 * must open its own xs handle and use it directly, rather than via
 * the libxl event machinery.

> I suppose we don't need to call the postfork-noexecing handler because
> we are inside libxl?

No, we don't need to call it because we're fast ("exit ... soon",
above).  The worst case is that we hold onto unwanted fds while the
openpty helper runs etc.

Ian.

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure"):
> On Wed, 2012-04-11 at 12:04 +0100, Ian Jackson wrote:
> > NB that libxl__ptr_add needs to be rewritten not to be quadratic in
> > the number of pointrs added (!)
> 
> Isn't it O(N) in numbers of pointers?

Yes, each addition is O(N).  Adding N pointers is O(N^2).

> I also just noticed that the initial:
>    /* fast case: we have space in the array for storing the pointer */
>     for (i = 0; i < gc->alloc_maxsize; i++) {
>         if (!gc->alloc_ptrs[i]) {
>             gc->alloc_ptrs[i] = ptr;
>             return 0;
>         }
>     }
> is a bit suboptimal since we never remove a ptr from the array, so we
> may as well track the max actually used. Then the fast path might
> actually be fast...

I thought we used to have a function for removing pointers from the gc
but I see that it has gone.  So yes this could be rewritten fairly
simply.

Ian.

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Jackson]

Ian Jackson writes ("Re: [Xen-devel] [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure"):
> I thought we used to have a function for removing pointers from the gc
> but I see that it has gone.  So yes this could be rewritten fairly
> simply.

Yes, libxl_free, removed on 16 August 2010.

Ian.

Re: [PATCH 29/31] libxl: ao: Convert libxl_run_bootloader

[Ian Campbell]

[...]
> +static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op);
> +static void bootloader_keystrokes_copyfail(libxl__egc *egc,
> +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> +static void bootloader_display_copyfail(libxl__egc *egc,
> +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> +static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
> +                                pid_t pid, int status);

I suppose that in here there is a sequence of callbacks which get
chained together? Might be worth pulling those out and ordering them in
the normal expected chain of execution?

[...]
> +/*----- synchronous subroutines -----*/
> +
> +static int setup_xenconsoled_pty(libxl__egc *egc, libxl__bootloader_state *bl,
> +                                 char *slave_path, size_t slave_path_len)
>  {
> +    STATE_AO_GC(bl);

I wondered about this in the patch which defined it but didn't really
consider it til I saw this user.

I didn't quite realise back then that STATE_AO_GC takes a pointer to a
struct which is user (a libxl internal user, but still) defined and
requires that it has a member "ao". I don't mind that for cases where
the struct is defined alongside the macro as part of a while API but for
separately defined structs its a bit nasty. Could it be:
	STATE_AO_GC(bl->ao)
instead?

I suppose that isn't really a criticism of this patch but rather of the
patch which added the macro, so this patch in itself is:

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Re: [PATCH 28/31] libxl: provide libxl__openpty_*

[Ian Campbell]

On Wed, 2012-04-11 at 12:19 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 28/31] libxl: provide libxl__openpty_*"):
> > On Tue, 2012-04-10 at 20:08 +0100, Ian Jackson wrote:
> > > General facility for ao operations to open ptys.
> ...
> > > +    if (!pid) {
> > > +        /* child */
> > > +        close(sockets[0]);
> > > +        signal(SIGCHLD, SIG_DFL);
> > > +
> > > +        for (i=0; i<count; i++) {
> > > +            r = openpty(&ptyfds[i][0], &ptyfds[i][1], NULL, termp, winp);
> > > +            if (r) { LOGE(ERROR,"openpty failed"); _exit(-1); }
> > > +        }
> > > +        rc = libxl__sendmsg_fds(gc, sockets[1], "",1,
> > > +                                2*count, &ptyfds[0][0], "ptys");
> > > +        if (rc) { LOGE(ERROR,"sendmsg to parent failed"); _exit(-1); }
> > 
> > Buried in this LOGE is a CTX somewhere, right? Is it valid to keep using
> > it? Perhaps it's OK just for this logging.
> 
> This should be documented somewhere.  But I see it isn't.  The closest
> we have is:
> 
>   * The child should go on to exec (or exit) soon, and should not make
>   * any further libxl event calls in the meantime.
> 
> I have clarified this:
> 
>  * The child should go on to exec (or exit) soon.  The child may not
>  * make any further calls to libxl infrastructure, except for memory
>  * allocation and logging.  If the child needs to use xenstore it
>  * must open its own xs handle and use it directly, rather than via
>  * the libxl event machinery.

Sounds good. I think I had mentally omitted the "event" in the original
too.

> > I suppose we don't need to call the postfork-noexecing handler because
> > we are inside libxl?
> 
> No, we don't need to call it because we're fast ("exit ... soon",
> above).  The worst case is that we hold onto unwanted fds while the
> openpty helper runs etc.

Oh, yes, of course.

Re: [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure

[Ian Campbell]

On Wed, 2012-04-11 at 12:21 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/31] libxl: Crash (more sensibly) on malloc failure"):
> > On Wed, 2012-04-11 at 12:04 +0100, Ian Jackson wrote:
> > > NB that libxl__ptr_add needs to be rewritten not to be quadratic in
> > > the number of pointrs added (!)
> > 
> > Isn't it O(N) in numbers of pointers?
> 
> Yes, each addition is O(N).  Adding N pointers is O(N^2).

Oh, right, yes.

Re: [PATCH 29/31] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 29/31] libxl: ao: Convert libxl_run_bootloader"):
> [...]
> > +static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op);
> > +static void bootloader_keystrokes_copyfail(libxl__egc *egc,
> > +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> > +static void bootloader_display_copyfail(libxl__egc *egc,
> > +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> > +static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
> > +                                pid_t pid, int status);
> 
> I suppose that in here there is a sequence of callbacks which get
> chained together? Might be worth pulling those out and ordering them in
> the normal expected chain of execution?

Yes.  The two "normal" callbacks are gotptys and finished, which
happen in that order.  The copyfail callbacks may happen in between.
So these are already in execution order.

> I wondered about this in the patch which defined it but didn't really
> consider it til I saw this user.
> 
> I didn't quite realise back then that STATE_AO_GC takes a pointer to a
> struct which is user (a libxl internal user, but still) defined and
> requires that it has a member "ao".

This seemed a fine restriction to me, but I'm not wedded to it.

> I don't mind that for cases where
> the struct is defined alongside the macro as part of a while API but for
> separately defined structs its a bit nasty. Could it be:
> 	STATE_AO_GC(bl->ao)
> instead?

So I will do this.  (I have removed your ack from my copy of that
underlying patch.)

> I suppose that isn't really a criticism of this patch but rather of the
> patch which added the macro, so this patch in itself is:

Thanks.

Ian.

Re: [PATCH 29/31] libxl: ao: Convert libxl_run_bootloader

[Ian Campbell]

On Wed, 2012-04-11 at 12:39 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 29/31] libxl: ao: Convert libxl_run_bootloader"):
> > [...]
> > > +static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op);
> > > +static void bootloader_keystrokes_copyfail(libxl__egc *egc,
> > > +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> > > +static void bootloader_display_copyfail(libxl__egc *egc,
> > > +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> > > +static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
> > > +                                pid_t pid, int status);
> > 
> > I suppose that in here there is a sequence of callbacks which get
> > chained together? Might be worth pulling those out and ordering them in
> > the normal expected chain of execution?
> 
> Yes.  The two "normal" callbacks are gotptys and finished, which
> happen in that order.  The copyfail callbacks may happen in between.
> So these are already in execution order.

Oh, good then. I suppose that proves that ordering them that way is not
as helpful as I thought it would be ;-)

> > I wondered about this in the patch which defined it but didn't really
> > consider it til I saw this user.
> > 
> > I didn't quite realise back then that STATE_AO_GC takes a pointer to a
> > struct which is user (a libxl internal user, but still) defined and
> > requires that it has a member "ao".
> 
> This seemed a fine restriction to me, but I'm not wedded to it.
> 
> > I don't mind that for cases where
> > the struct is defined alongside the macro as part of a while API but for
> > separately defined structs its a bit nasty. Could it be:
> > 	STATE_AO_GC(bl->ao)
> > instead?
> 
> So I will do this.  (I have removed your ack from my copy of that
> underlying patch.)

Thanks, sorry for prematurely Acking that patch.

FWIW, with the change above you could put it back...

> 
> > I suppose that isn't really a criticism of this patch but rather of the
> > patch which added the macro, so this patch in itself is:
> 
> Thanks.
> 
> Ian.