(no subject)

(no subject)

[Ian Jackson]

>From Ian Jackson <ian.jackson@eu.citrix.com> # This line is ignored.
From: Ian Jackson <ian.jackson@eu.citrix.com>
Subject: [RFC PATCH 0/6] libxl: New child process handling
In-Reply-To: 

This is the first draft of my new libxl API for child process
handling.

 1/6 libxl: Fix leak of ctx->lock
 2/6 libxl: abolish libxl_ctx_postfork
 3/6 tools: Correct PTHREAD options in config/StdGNU.mk
 4/6 libxl: Protect fds with CLOEXEC even with forking threads
 5/6 libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS
 6/6 libxl: event API: new facilities for waiting for subprocesses

Please comment.  I'm particularly keen on comments about:

 * The correctness of the pthread_atfork logic in 4/6.
 * The sufficiency for applications of the child handling API in 6/6.
 * The portability of the pthread command line option changes in 3/6.
 * Correctness :-).

If you want to apply it to make sense of, you'll need to know that
this series is on top of my recently-posted 3-patch libxl event fixup
series.

Thanks,
Ian.

[PATCH 1/6] libxl: Fix leak of ctx->lock

[Ian Jackson]

A mutex created with pthread_mutex_init, like ctx->lock, may need to
be destroyed with pthread_mutex_destroy.

Also, previously, if libxl__init_recursive_mutex failed, the nascent
ctx would be leaked.  Add some comments which will hopefully make
these kind of mistakes less likely in future.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.c |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 7735223..fd890cf 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -39,10 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     memset(ctx, 0, sizeof(libxl_ctx));
     ctx->lg = lg;
 
-    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
-        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
-        return ERROR_FAIL;
-    }
+    /* First initialise pointers (cannot fail) */
 
     LIBXL_TAILQ_INIT(&ctx->occurred);
 
@@ -61,6 +58,16 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     LIBXL_TAILQ_INIT(&ctx->death_list);
     libxl__ev_xswatch_init(&ctx->death_watch);
 
+    /* The mutex is special because we can't idempotently destroy it */
+
+    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
+        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
+        free(ctx);
+        ctx = 0;
+    }
+
+    /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
+
     rc = libxl__poller_init(ctx, &ctx->poller_app);
     if (rc) goto out;
 
@@ -150,6 +157,8 @@ int libxl_ctx_free(libxl_ctx *ctx)
 
     discard_events(&ctx->occurred);
 
+    pthread_mutex_destroy(&ctx->lock);
+
     GC_FREE;
     free(ctx);
     return 0;
-- 
1.7.2.5

Re: [PATCH 1/6] libxl: Fix leak of ctx->lock

[Ian Campbell]

On Fri, 2012-02-17 at 19:15 +0000, Ian Jackson wrote:
> A mutex created with pthread_mutex_init, like ctx->lock, may need to
> be destroyed with pthread_mutex_destroy.
> 
> Also, previously, if libxl__init_recursive_mutex failed, the nascent
> ctx would be leaked.  Add some comments which will hopefully make
> these kind of mistakes less likely in future.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl.c |   17 +++++++++++++----
>  1 files changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 7735223..fd890cf 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -39,10 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
>      memset(ctx, 0, sizeof(libxl_ctx));
>      ctx->lg = lg;
>  
> -    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
> -        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
> -        return ERROR_FAIL;
> -    }
> +    /* First initialise pointers (cannot fail) */
>  
>      LIBXL_TAILQ_INIT(&ctx->occurred);
>  
> @@ -61,6 +58,16 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
>      LIBXL_TAILQ_INIT(&ctx->death_list);
>      libxl__ev_xswatch_init(&ctx->death_watch);
>  
> +    /* The mutex is special because we can't idempotently destroy it */
> +
> +    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
> +        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
> +        free(ctx);
> +        ctx = 0;
> +    }
> +
> +    /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
> +
>      rc = libxl__poller_init(ctx, &ctx->poller_app);
>      if (rc) goto out;
>  
> @@ -150,6 +157,8 @@ int libxl_ctx_free(libxl_ctx *ctx)
>  
>      discard_events(&ctx->occurred);
>  
> +    pthread_mutex_destroy(&ctx->lock);
> +
>      GC_FREE;
>      free(ctx);
>      return 0;

[PATCH 2/6] libxl: abolish libxl_ctx_postfork

[Ian Jackson]

libxl's task has become too complicated (particularly in the presence
of both forking and multithreading) to support reuse of the same
libxl_ctx after fork.

So abolish libxl_ctx_fork.  xl instead simply initialises a new
libxl_ctx.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.h       |    1 -
 tools/libxl/libxl_utils.c |    7 -------
 tools/libxl/xl.c          |    8 ++++++++
 tools/libxl/xl.h          |    1 +
 tools/libxl/xl_cmdimpl.c  |    8 ++------
 5 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 1bffa03..19ef1de 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -312,7 +312,6 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
                     unsigned flags /* none currently defined */,
                     xentoollog_logger *lg);
 int libxl_ctx_free(libxl_ctx *ctx /* 0 is OK */);
-int libxl_ctx_postfork(libxl_ctx *ctx);
 
 /* domain related functions */
 int libxl_init_create_info(libxl_ctx *ctx, libxl_domain_create_info *c_info);
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index cd819c8..9f9f91d 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -399,13 +399,6 @@ READ_WRITE_EXACTLY(read, 1, /* */)
 READ_WRITE_EXACTLY(write, 0, const)
 
 
-int libxl_ctx_postfork(libxl_ctx *ctx) {
-    if (ctx->xsh) xs_daemon_destroy_postfork(ctx->xsh);
-    ctx->xsh = xs_daemon_open();
-    if (!ctx->xsh) return ERROR_FAIL;
-    return 0;
-}
-
 pid_t libxl_fork(libxl_ctx *ctx)
 {
     pid_t pid;
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index df9b1e7..9fd67b4 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -95,6 +95,14 @@ static void parse_global_config(const char *configfile,
     xlu_cfg_destroy(config);
 }
 
+void postfork(void)
+{
+    if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
+        fprintf(stderr, "cannot reinit xl context after fork\n");
+        exit(-1);
+    }
+}
+
 int main(int argc, char **argv)
 {
     int opt = 0;
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index 702b208..394a128 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -104,6 +104,7 @@ struct cmd_spec *cmdtable_lookup(const char *s);
 
 extern libxl_ctx *ctx;
 extern xentoollog_logger_stdiostream *logger;
+void postfork(void);
 
 /* global options */
 extern int autoballoon;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index c58e9f3..2018153 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1422,7 +1422,7 @@ static int autoconnect_console(libxl_ctx *ctx, uint32_t domid, void *priv)
     } else if (*pid > 0)
         return 0;
 
-    libxl_ctx_postfork(ctx);
+    postfork();
 
     sleep(1);
     libxl_primary_console_exec(ctx, domid);
@@ -1709,11 +1709,7 @@ start:
             goto out;
         }
 
-        rc = libxl_ctx_postfork(ctx);
-        if (rc) {
-            LOG("failed to reinitialise context after fork");
-            exit(-1);
-        }
+        postfork();
 
         if (asprintf(&name, "xl-%s", d_config.c_info.name) < 0) {
             LOG("Failed to allocate memory in asprintf");
-- 
1.7.2.5

Re: [PATCH 2/6] libxl: abolish libxl_ctx_postfork

[Ian Campbell]

On Fri, 2012-02-17 at 19:15 +0000, Ian Jackson wrote:
> libxl's task has become too complicated (particularly in the presence
> of both forking and multithreading) to support reuse of the same
> libxl_ctx after fork.
> 
> So abolish libxl_ctx_fork.  xl instead simply initialises a new
> libxl_ctx.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[PATCH 3/6] tools: Correct PTHREAD options in config/StdGNU.mk

[Ian Jackson]

It is not correct to say -lpthread.  The correct option is -pthread,
which may have sundry other effects on code generation etc.  It needs
to be passed both to compilation and linking.

So abolish PTHREAD_LIBS in StdGNU.mk, and add PTHREAD_CFLAGS and
PTHREAD_LDFLAGS instead.  Fix the one user (libxc).

There are still some other users in tree which pass -pthread or
-lpthread by adding it as a literal to their own compiler options.
These will be fixed in a later version of this patch.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 config/StdGNU.mk     |    4 +++-
 tools/libxc/Makefile |    4 +++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/config/StdGNU.mk b/config/StdGNU.mk
index 2af2841..97445a7 100644
--- a/config/StdGNU.mk
+++ b/config/StdGNU.mk
@@ -68,10 +68,12 @@ XEN_SCRIPT_DIR = $(XEN_CONFIG_DIR)/scripts
 
 SOCKET_LIBS =
 CURSES_LIBS = -lncurses
-PTHREAD_LIBS = -lpthread
 UTIL_LIBS = -lutil
 DLOPEN_LIBS = -ldl
 
+PTHREAD_CFLAGS = -pthread
+PTHREAD_LDFLAGS = -pthread
+
 SONAME_LDFLAG = -soname
 SHLIB_LDFLAGS = -shared
 
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index b5e7022..09e2f5f 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -72,6 +72,8 @@ CFLAGS   += -I. $(CFLAGS_xeninclude)
 # Needed for posix_fadvise64() in xc_linux.c
 CFLAGS-$(CONFIG_Linux) += -D_GNU_SOURCE
 
+CFLAGS	+= $(PTHREAD_CFLAGS)
+
 # Define this to make it possible to run valgrind on code linked with these
 # libraries.
 #CFLAGS   += -DVALGRIND -O0 -ggdb3
@@ -156,7 +158,7 @@ libxenctrl.so.$(MAJOR): libxenctrl.so.$(MAJOR).$(MINOR)
 	ln -sf $< $@
 
 libxenctrl.so.$(MAJOR).$(MINOR): $(CTRL_PIC_OBJS)
-	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
+	$(CC) $(LDFLAGS) $(PTHREAD_LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
 
 # libxenguest
 
-- 
1.7.2.5

[PATCH 4/6] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Jackson]

We introduce a new "carefd" concept, which relates to fds that we care
about not being inherited by long-lived children.

As yet we do not use this anywherein libxl.  Until all locations in
libxl which make such fds are converted, libxl__postfork may not work
entirely properly.  If these locations do not use O_CLOEXEC (or use
calls for which there is no O_CLOEXEC) then multithreaded programs may
not work properly.

This introduces a new API call libxl_postfork_child_noexec which must
be called by applications which make long-running non-execing
children.  Add the appropriate call to xl's postfork function.

On Linux pthread_atfork demands compilation with -pthread, so add
PTHREAD_CFLAGS, _LDFLAGS, _LIBS to the corresponding variables in the
libxl Makefile.  It is not clear why we got away without these before.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/Makefile         |    6 +-
 tools/libxl/libxl.c          |    3 +
 tools/libxl/libxl_event.h    |   10 +++
 tools/libxl/libxl_fork.c     |  167 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   65 ++++++++++++++++
 tools/libxl/xl.c             |    3 +
 6 files changed, 253 insertions(+), 1 deletions(-)
 create mode 100644 tools/libxl/libxl_fork.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 06764f2..a041294 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -26,6 +26,10 @@ endif
 LIBXL_LIBS =
 LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(UTIL_LIBS) $(LIBUUID_LIBS)
 
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
+LIBXL_LIBS += $(PTHREAD_LIBS)
+
 LIBXLU_LIBS =
 
 LIBXL_OBJS-y = osdeps.o libxl_paths.o libxl_bootloader.o flexarray.o
@@ -53,7 +57,7 @@ LIBXL_LIBS += -lyajl
 LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
 			libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
 			libxl_internal.o libxl_utils.o libxl_uuid.o libxl_json.o \
-			libxl_qmp.o libxl_event.o $(LIBXL_OBJS-y)
+			libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
 LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
 
 $(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore) $(CFLAGS_libblktapctl)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index fd890cf..716163a 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -68,6 +68,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
 
     /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
 
+    rc = libxl__atfork_init(ctx);
+    if (rc) goto out;
+
     rc = libxl__poller_init(ctx, &ctx->poller_app);
     if (rc) goto out;
 
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index f889115..74a0402 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -369,6 +369,16 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
  */
 void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 
+
+/*
+ * An application which initialises a libxl_ctx in a parent process
+ * and then forks a child which does not quickly exec, must
+ * instead libxl__postfork_child_noexec in the child.  One call
+ * for all libxl contexts is sufficient.
+ */
+void libxl_postfork_child_noexec(libxl_ctx *ctx);
+
+
 #endif
 
 /*
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
new file mode 100644
index 0000000..66d0ee0
--- /dev/null
+++ b/tools/libxl/libxl_fork.c
@@ -0,0 +1,167 @@
+/*
+ * Copyright (C) 2012      Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+/*
+ * Internal child process machinery for use by other parts of libxl
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*
+ * carefd arrangements
+ *
+ * carefd_begin and _unlock take out the no_forking lock, which we
+ * also take and release in our pthread_atfork handlers.  So when this
+ * lock is held the whole process cannot fork.  We therefore protect
+ * our fds from leaking into children made by other threads.
+ *
+ * We maintain a list of all the carefds, so that if the application
+ * wants to fork a long-running but non-execing child, we can close
+ * them all.
+ *
+ * So the record function sets CLOEXEC for the benefit of execing
+ * children, and makes a note of the fd for the benefit of non-execing
+ * ones.
+ */
+
+struct libxl__carefd {
+    LIBXL_LIST_ENTRY(libxl__carefd) entry;
+    int fd;
+};
+
+static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;
+static int atfork_registered;
+static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
+    LIBXL_LIST_HEAD_INITIALIZER(carefds);
+
+static void atfork_lock(void)
+{
+    int r = pthread_mutex_lock(&no_forking);
+    assert(!r);
+}
+
+static void atfork_unlock(void)
+{
+    int r = pthread_mutex_unlock(&no_forking);
+    assert(!r);
+}
+
+int libxl__atfork_init(libxl_ctx *ctx)
+{
+    int r, rc;
+    
+    atfork_lock();
+    if (atfork_registered)
+        return 0;
+
+    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
+    if (r) {
+        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "pthread_atfork failed");
+        rc = ERROR_NOMEM;
+        goto out;
+    }
+
+    atfork_registered = 1;
+    rc = 0;
+ out:
+    atfork_unlock();
+    return rc;
+}
+
+void libxl__carefd_begin(void) { atfork_lock(); }
+void libxl__carefd_unlock(void) { atfork_unlock(); }
+
+libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
+{
+    libxl__carefd *cf = 0;
+    int rc;
+
+    if (fd >= 0) {
+        rc = libxl_fd_set_cloexec(ctx, fd, 1);
+        if (rc) goto out;
+    }
+
+    cf = malloc(sizeof(*cf));
+    if (!cf) {
+        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "failed to malloc for carefd");
+        goto out;
+    }
+    cf->fd = fd;
+    LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry);
+    return cf;
+
+ out:
+    if (cf) free(cf);
+    return 0;
+}
+
+libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd)
+{
+    libxl__carefd *cf = 0;
+    int r;
+
+    cf = libxl__carefd_record(ctx, fd);
+    if (!cf) {
+        r = close(fd);
+        if (r)
+            /* Carry on as best we can. */
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING, "failed to close new fd");
+    }
+    libxl__carefd_unlock();
+    return cf;
+}
+
+void libxl_postfork_child_noexec(libxl_ctx *ctx)
+{
+    libxl__carefd *cf, *cf_tmp;
+    int r;
+
+    atfork_lock();
+    LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
+        if (cf->fd >= 0) {
+            r = close(cf->fd);
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING, "failed to close fd=%d"
+                             " (perhaps of another libxl ctx)", cf->fd);
+        }
+        free(cf);
+    }
+    LIBXL_LIST_INIT(&carefds);
+    atfork_unlock();
+}
+
+int libxl__carefd_close(libxl__carefd *cf)
+{
+    if (!cf) return 0;
+    int r = cf->fd < 0 ? 0 : close(cf->fd);
+    int esave = errno;
+    LIBXL_LIST_REMOVE(cf, entry);
+    free(cf);
+    errno = esave;
+    return r;
+}
+
+int libxl__carefd_fd(const libxl__carefd *cf)
+{
+    if (!cf) return -1;
+    return cf->fd;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 7b8d0c2..088a417 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -588,6 +588,9 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
 
 
+int libxl__atfork_init(libxl_ctx *ctx);
+
+
 /* from xl_dom */
 _hidden libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid);
 _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
@@ -1267,6 +1270,68 @@ _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
 /* For use by ao machinery ONLY */
 _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
 
+
+/*
+ * File descriptors and CLOEXEC
+ */
+
+/*
+ * For libxl functions which create file descriptors, at least one
+ * of the following must be true:
+ *  (a) libxl does not care if copies of this open-file are inherited
+ *      by random children and might remain open indefinitely
+ *  (b) libxl must take extra care for the fd (the actual descriptor,
+ *      not the open-file) as below.  We call this a "carefd".
+ *
+ * The rules for opening a carefd are:
+ *  (i)   Before bringing any carefds into existence,
+ *        libxl code must call libxl__carefd_begin.
+ *  (ii)  Then for each carefd brought into existence,
+ *        libxl code must call libxl__carefd_record
+ *        and remember the libxl__carefd_record*.
+ *  (iii) Then it must call libxl__carefd_unlock.
+ *  (iv)  When in a child process the fd is to be passed across
+ *        exec by libxl, the libxl code must unset FD_CLOEXEC
+ *        on the fd eg by using libxl_fd_set_cloexec.
+ *  (v)   Later, when the fd is to be closed in the same process,
+ *        libxl code must not call close.  Instead, it must call
+ *        libxl__carefd_close.
+ * Steps (ii) and (iii) can be combined by calling the convenience
+ * function libxl__carefd_opened.
+ */
+/* libxl__carefd_begin and _unlock (or _opened) must be called always
+ * in pairs.  They may be called with the CTX lock held.  In between
+ * _begin and _unlock, the following are prohibited:
+ *   - anything which might block
+ *   - any callbacks to the application
+ *   - nested calls to libxl__carefd_begin
+ *   - fork (libxl__fork)
+ * In general nothing should be done before _unlock that could be done
+ * afterwards.
+ */
+typedef struct libxl__carefd libxl__carefd;
+
+void libxl__carefd_begin(void);
+void libxl__carefd_unlock(void);
+
+/* fd may be -1, in which case this returns a dummy libxl__fd_record
+ * on which it _carefd_close is a no-op.  May fail due to lack of memory
+ * in which case it will have logged and will return NULL. */
+libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd);
+
+libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd);
+/* Combines _record and _unlock in a single call.  If fd==-1,
+ * still does the unlock, but returns 0.  If it fails and fd!=-1,
+ * it will close the fd before returning. */
+
+/* Works just like close(2).  You may pass NULL, in which case it's
+ * a successful no-op. */
+int libxl__carefd_close(libxl__carefd*);
+
+/* You may pass NULL in which case the answer is -1. */
+int libxl__carefd_fd(const libxl__carefd*);
+
+
 /*
  * Convenience macros.
  */
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index 9fd67b4..d806077 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -97,6 +97,9 @@ static void parse_global_config(const char *configfile,
 
 void postfork(void)
 {
+    libxl_postfork_child_noexec(ctx); /* in case we don't exit/exec */
+    ctx = 0;
+
     if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
         fprintf(stderr, "cannot reinit xl context after fork\n");
         exit(-1);
-- 
1.7.2.5

Re: [PATCH 4/6] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Campbell]

On Fri, 2012-02-17 at 19:15 +0000, Ian Jackson wrote:
> We introduce a new "carefd" concept, which relates to fds that we care
> about not being inherited by long-lived children.
> 
> As yet we do not use this anywherein libxl.  Until all locations in

"anywherein"

> libxl which make such fds are converted, libxl__postfork may not work
> entirely properly.  If these locations do not use O_CLOEXEC (or use
> calls for which there is no O_CLOEXEC) then multithreaded programs may
> not work properly.
> 
> This introduces a new API call libxl_postfork_child_noexec which must
> be called by applications which make long-running non-execing
> children.  Add the appropriate call to xl's postfork function.
> 
> On Linux pthread_atfork demands compilation with -pthread, so add
> PTHREAD_CFLAGS, _LDFLAGS, _LIBS to the corresponding variables in the
> libxl Makefile.  It is not clear why we got away without these before.

Since I assume the Makefile bit could safely be split out and applied,
that bit is:
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>

> diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
> index f889115..74a0402 100644
> --- a/tools/libxl/libxl_event.h
> +++ b/tools/libxl/libxl_event.h
> @@ -369,6 +369,16 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
>   */
>  void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
> 
> +
> +/*
> + * An application which initialises a libxl_ctx in a parent process
> + * and then forks a child which does not quickly exec, must
> + * instead libxl__postfork_child_noexec in the child.  One call
> + * for all libxl contexts is sufficient.
> + */
> +void libxl_postfork_child_noexec(libxl_ctx *ctx);

Is the ctx passed here required to be one of the existing ctx's or a
newly allocated one in this context?

> +
> +
>  #endif
> 
>  /*
> diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
> new file mode 100644
> index 0000000..66d0ee0
> --- /dev/null
> +++ b/tools/libxl/libxl_fork.c
> @@ -0,0 +1,167 @@
> +/*
> + * Copyright (C) 2012      Citrix Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2.1 only. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + */
> +/*
> + * Internal child process machinery for use by other parts of libxl
> + */
> +
> +#include "libxl_osdeps.h" /* must come before any other headers */
> +
> +#include "libxl_internal.h"
> +
> +/*
> + * carefd arrangements
> + *
> + * carefd_begin and _unlock take out the no_forking lock, which we
> + * also take and release in our pthread_atfork handlers.  So when this
> + * lock is held the whole process cannot fork.  We therefore protect
> + * our fds from leaking into children made by other threads.
> + *
> + * We maintain a list of all the carefds, so that if the application
> + * wants to fork a long-running but non-execing child, we can close
> + * them all.
> + *
> + * So the record function sets CLOEXEC for the benefit of execing
> + * children, and makes a note of the fd for the benefit of non-execing
> + * ones.
> + */
> +
> +struct libxl__carefd {
> +    LIBXL_LIST_ENTRY(libxl__carefd) entry;
> +    int fd;
> +};
> +
> +static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;

We previously had trouble with PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP
not being defined on NetBSD.

http://www.daemon-systems.org/man/pthread_mutex_init.3.html suggests
PTHREAD_MUTEX_INITIALIZER is but perhaps we should confirm?

> +static int atfork_registered;

Does pthread_atfork persist into the children?

If the parent has set this then it will be set for the child too, which
if the answer to my question is "No" is a problem.

I suspect the answer is yes though.

> +static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
> +    LIBXL_LIST_HEAD_INITIALIZER(carefds);
> +
> +static void atfork_lock(void)
> +{
> +    int r = pthread_mutex_lock(&no_forking);
> +    assert(!r);
> +}
> +
> +static void atfork_unlock(void)
> +{
> +    int r = pthread_mutex_unlock(&no_forking);
> +    assert(!r);
> +}
> +
> +int libxl__atfork_init(libxl_ctx *ctx)
> +{
> +    int r, rc;
> +
> +    atfork_lock();
> +    if (atfork_registered)

Missing an unlock?

> +        return 0;
> +
> +    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
> +    if (r) {
> +        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "pthread_atfork failed");
> +        rc = ERROR_NOMEM;
> +        goto out;
> +    }
> +
> +    atfork_registered = 1;
> +    rc = 0;
> + out:
> +    atfork_unlock();
> +    return rc;
> +}
> +
> +void libxl__carefd_begin(void) { atfork_lock(); }
> +void libxl__carefd_unlock(void) { atfork_unlock(); }

This naming seems asymmetric.

Ian.

Re: [PATCH 4/6] libxl: Protect fds with CLOEXEC even with forking threads

[Roger Pau Monné]

2012/2/20 Ian Campbell <Ian.Campbell@citrix.com>:
> On Fri, 2012-02-17 at 19:15 +0000, Ian Jackson wrote:
>> We introduce a new "carefd" concept, which relates to fds that we care
>> about not being inherited by long-lived children.
>>
>> As yet we do not use this anywherein libxl.  Until all locations in
>
> "anywherein"
>
>> libxl which make such fds are converted, libxl__postfork may not work
>> entirely properly.  If these locations do not use O_CLOEXEC (or use
>> calls for which there is no O_CLOEXEC) then multithreaded programs may
>> not work properly.
>>
>> This introduces a new API call libxl_postfork_child_noexec which must
>> be called by applications which make long-running non-execing
>> children.  Add the appropriate call to xl's postfork function.
>>
>> On Linux pthread_atfork demands compilation with -pthread, so add
>> PTHREAD_CFLAGS, _LDFLAGS, _LIBS to the corresponding variables in the
>> libxl Makefile.  It is not clear why we got away without these before.
>
> Since I assume the Makefile bit could safely be split out and applied,
> that bit is:
> Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
>
>> diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
>> index f889115..74a0402 100644
>> --- a/tools/libxl/libxl_event.h
>> +++ b/tools/libxl/libxl_event.h
>> @@ -369,6 +369,16 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
>>   */
>>  void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
>>
>> +
>> +/*
>> + * An application which initialises a libxl_ctx in a parent process
>> + * and then forks a child which does not quickly exec, must
>> + * instead libxl__postfork_child_noexec in the child.  One call
>> + * for all libxl contexts is sufficient.
>> + */
>> +void libxl_postfork_child_noexec(libxl_ctx *ctx);
>
> Is the ctx passed here required to be one of the existing ctx's or a
> newly allocated one in this context?
>
>> +
>> +
>>  #endif
>>
>>  /*
>> diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
>> new file mode 100644
>> index 0000000..66d0ee0
>> --- /dev/null
>> +++ b/tools/libxl/libxl_fork.c
>> @@ -0,0 +1,167 @@
>> +/*
>> + * Copyright (C) 2012      Citrix Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU Lesser General Public License as published
>> + * by the Free Software Foundation; version 2.1 only. with the special
>> + * exception on linking described in file LICENSE.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU Lesser General Public License for more details.
>> + */
>> +/*
>> + * Internal child process machinery for use by other parts of libxl
>> + */
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +
>> +/*
>> + * carefd arrangements
>> + *
>> + * carefd_begin and _unlock take out the no_forking lock, which we
>> + * also take and release in our pthread_atfork handlers.  So when this
>> + * lock is held the whole process cannot fork.  We therefore protect
>> + * our fds from leaking into children made by other threads.
>> + *
>> + * We maintain a list of all the carefds, so that if the application
>> + * wants to fork a long-running but non-execing child, we can close
>> + * them all.
>> + *
>> + * So the record function sets CLOEXEC for the benefit of execing
>> + * children, and makes a note of the fd for the benefit of non-execing
>> + * ones.
>> + */
>> +
>> +struct libxl__carefd {
>> +    LIBXL_LIST_ENTRY(libxl__carefd) entry;
>> +    int fd;
>> +};
>> +
>> +static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;
>
> We previously had trouble with PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP
> not being defined on NetBSD.
>
> http://www.daemon-systems.org/man/pthread_mutex_init.3.html suggests
> PTHREAD_MUTEX_INITIALIZER is but perhaps we should confirm?


Just tested on a NetBSD system, and PTHREAD_MUTEX_INITIALIZER macro is allowed.

>> +static int atfork_registered;
>
> Does pthread_atfork persist into the children?
>
> If the parent has set this then it will be set for the child too, which
> if the answer to my question is "No" is a problem.
>
> I suspect the answer is yes though.
>
>> +static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
>> +    LIBXL_LIST_HEAD_INITIALIZER(carefds);
>> +
>> +static void atfork_lock(void)
>> +{
>> +    int r = pthread_mutex_lock(&no_forking);
>> +    assert(!r);
>> +}
>> +
>> +static void atfork_unlock(void)
>> +{
>> +    int r = pthread_mutex_unlock(&no_forking);
>> +    assert(!r);
>> +}
>> +
>> +int libxl__atfork_init(libxl_ctx *ctx)
>> +{
>> +    int r, rc;
>> +
>> +    atfork_lock();
>> +    if (atfork_registered)
>
> Missing an unlock?
>
>> +        return 0;
>> +
>> +    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
>> +    if (r) {
>> +        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "pthread_atfork failed");
>> +        rc = ERROR_NOMEM;
>> +        goto out;
>> +    }
>> +
>> +    atfork_registered = 1;
>> +    rc = 0;
>> + out:
>> +    atfork_unlock();
>> +    return rc;
>> +}
>> +
>> +void libxl__carefd_begin(void) { atfork_lock(); }
>> +void libxl__carefd_unlock(void) { atfork_unlock(); }
>
> This naming seems asymmetric.
>
> Ian.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH 4/6] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 4/6] libxl: Protect fds with CLOEXEC even with forking threads"):
> On Fri, 2012-02-17 at 19:15 +0000, Ian Jackson wrote:
> > As yet we do not use this anywherein libxl.  Until all locations in
> 
> "anywherein"

Fixed.

> > libxl which make such fds are converted, libxl__postfork may not work
> > entirely properly.  If these locations do not use O_CLOEXEC (or use
> > calls for which there is no O_CLOEXEC) then multithreaded programs may
> > not work properly.
> > 
> > This introduces a new API call libxl_postfork_child_noexec which must
> > be called by applications which make long-running non-execing
> > children.  Add the appropriate call to xl's postfork function.
> > 
> > On Linux pthread_atfork demands compilation with -pthread, so add
> > PTHREAD_CFLAGS, _LDFLAGS, _LIBS to the corresponding variables in the
> > libxl Makefile.  It is not clear why we got away without these before.
> 
> Since I assume the Makefile bit could safely be split out and applied,
> that bit is:
> Acked-by: Ian Campbell <Ian.Campbell@citrix.com>

I'll split it out into a separate patch.

> > +/*
> > + * An application which initialises a libxl_ctx in a parent process
> > + * and then forks a child which does not quickly exec, must
> > + * instead libxl__postfork_child_noexec in the child.  One call
> > + * for all libxl contexts is sufficient.
> > + */
> > +void libxl_postfork_child_noexec(libxl_ctx *ctx);
> 
> Is the ctx passed here required to be one of the existing ctx's or a
> newly allocated one in this context?

I will clarify this comment.

> > +static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;
> 
> We previously had trouble with PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP
> not being defined on NetBSD.
> 
> http://www.daemon-systems.org/man/pthread_mutex_init.3.html suggests
> PTHREAD_MUTEX_INITIALIZER is but perhaps we should confirm?

PTHREAD_MUTEX_INITIALIZER is in POSIX and the _NP one isn't.  But I
see Roger has confirmed it's on BSD too, which is good.

> > +static int atfork_registered;
> 
> Does pthread_atfork persist into the children?
> 
> If the parent has set this then it will be set for the child too, which
> if the answer to my question is "No" is a problem.
> 
> I suspect the answer is yes though.

Yes.  (Although I can't find any explicit statement, I think it's
obvious.)

> > +    atfork_lock();
> > +    if (atfork_registered)
> > +        return 0;
> 
> Missing an unlock?

Silly me, I should be using "goto out" as that's what it's for.

> > +void libxl__carefd_begin(void) { atfork_lock(); }
> > +void libxl__carefd_unlock(void) { atfork_unlock(); }
> 
> This naming seems asymmetric.

It's not a symmetric situation.  The complete sequence is one of these
three:

   libxl__carefd_begin
   libxl__carefd_record
   libxl__carefd_unlock
   ...
   libxl__carefd_close

or

   libxl__carefd_begin
   libxl__carefd_opened
   ...
   libxl__carefd_close

or

   libxl__carefd_begin
   libxl__carefd_unlock

I'm open to suggestions for alternative names.

Ian.

[PATCH 5/6] libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS

[Ian Jackson]

Introduce definition and use of a new function-local macro REQUIRE_FDS
to avoid repeatedly spelling out which fds we are interested in.

We are going to introduce a new fd for the SIGCHLD self-pipe.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_event.c |   82 ++++++++++++++++++++++++++++++--------------
 1 files changed, 56 insertions(+), 26 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 522bd99..8542e24 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -593,6 +593,45 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
     int rc;
 
     /*
+     * We need to look at the fds we want twice: firstly, to count
+     * them so we can make the rindex array big enough, and secondly
+     * to actually fill the arrays in.
+     *
+     * To ensure correctness and avoid repeating the logic for
+     * deciding which fds are relevant, we define a macro
+     *    REQUIRE_FDS( BODY )
+     * which calls
+     *    do{
+     *        int req_fd;
+     *        int req_events;
+     *        BODY;
+     *    }while(0)
+     * for each fd with a nonzero events.  This is invoked twice.
+     *
+     * The definition of REQUIRE_FDS is simplified with the helper
+     * macro
+     *    void REQUIRE_FD(int req_fd, int req_events, BODY);
+     */
+
+#define REQUIRE_FDS(BODY) do{                                          \
+                                                                       \
+        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)                     \
+            REQUIRE_FD(efd->fd, efd->events, BODY);                    \
+                                                                       \
+        REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
+                                                                       \
+    }while(0)
+
+#define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
+        int req_events = (req_events_);                 \
+        int req_fd = (req_fd_);                         \
+        if (req_events) {                               \
+            BODY;                                       \
+        }                                               \
+    }while(0)
+
+
+    /*
      * In order to be able to efficiently find the libxl__ev_fd
      * for a struct poll during _afterpoll, we maintain a shadow
      * data structure in CTX->fd_beforepolled: each slot in
@@ -609,13 +648,13 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
          * not to mess with fd_rindex.
          */
 
-        int maxfd = poller->wakeup_pipe[0] + 1;
-        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
-            if (!efd->events)
-                continue;
-            if (efd->fd >= maxfd)
-                maxfd = efd->fd + 1;
-        }
+        int maxfd = 0;
+
+        REQUIRE_FDS({
+            if (req_fd >= maxfd)
+                maxfd = req_fd + 1;
+        });
+
         /* make sure our array is as big as *nfds_io */
         if (poller->fd_rindex_allocd < maxfd) {
             assert(maxfd < INT_MAX / sizeof(int) / 2);
@@ -630,25 +669,16 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
 
     int used = 0;
 
-#define REQUIRE_FD(req_fd, req_events, efd) do{                 \
-        if ((req_events)) {                                     \
-            if (used < *nfds_io) {                              \
-                fds[used].fd = (req_fd);                        \
-                fds[used].events = (req_events);                \
-                fds[used].revents = 0;                          \
-                assert((req_fd) < poller->fd_rindex_allocd);    \
-                poller->fd_rindex[(req_fd)] = used;             \
-            }                                                   \
-            used++;                                             \
-        }                                                       \
-    }while(0)
-
-    LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)
-        REQUIRE_FD(efd->fd, efd->events, efd);
-
-    REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, 0);
-
-#undef REQUIRE_FD
+    REQUIRE_FDS({
+        if (used < *nfds_io) {
+            fds[used].fd = req_fd;
+            fds[used].events = req_events;
+            fds[used].revents = 0;
+            assert(req_fd < poller->fd_rindex_allocd);
+            poller->fd_rindex[req_fd] = used;
+        }
+        used++;
+    });
 
     rc = used <= *nfds_io ? 0 : ERROR_BUFFERFULL;
 
-- 
1.7.2.5

[PATCH 6/6] libxl: event API: new facilities for waiting for subprocesses

[Ian Jackson]

The current arrangements in libxl for spawning subprocesses have two
key problems: (i) they make unwarranted (and largely undocumented)
assumptions about the caller's use of subprocesses, (ii) they aren't
integrated into the event system and can't be made asynchronous etc.

So replace them with a new set of facilities.

Primarily, from the point of view of code inside libxl, this is
libxl__ev_child_fork which is both (a) a version of fork() and (b) an
event source which generates a callback when the child dies.

>From the point of view of the application, we fully document our use
of SIGCHLD.  The application can tell us whether it wants to own
SIGCHLD or not; if it does, it has to tell us about deaths of our
children.

Currently there are no callers in libxl which use these facilities.
All code in libxl which forks needs to be converted and libxl_fork
needse to be be abolished.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.c          |   17 +++-
 tools/libxl/libxl_event.c    |   52 +++++++--
 tools/libxl/libxl_event.h    |  142 ++++++++++++++++++++++++-
 tools/libxl/libxl_fork.c     |  242 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   61 ++++++++++-
 5 files changed, 494 insertions(+), 20 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 716163a..06ac064 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -39,7 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     memset(ctx, 0, sizeof(libxl_ctx));
     ctx->lg = lg;
 
-    /* First initialise pointers (cannot fail) */
+    /* First initialise pointers etc. (cannot fail) */
 
     LIBXL_TAILQ_INIT(&ctx->occurred);
 
@@ -58,6 +58,11 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     LIBXL_TAILQ_INIT(&ctx->death_list);
     libxl__ev_xswatch_init(&ctx->death_watch);
 
+    ctx->childproc_hooks = &libxl__childproc_default_hooks;
+    ctx->childproc_user = 0;
+        
+    ctx->sigchld_selfpipe[0] = -1;
+
     /* The mutex is special because we can't idempotently destroy it */
 
     if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
@@ -160,6 +165,16 @@ int libxl_ctx_free(libxl_ctx *ctx)
 
     discard_events(&ctx->occurred);
 
+    /* If we have outstanding children, then the application inherits
+     * them; we wish the application good luck with understanding
+     * this if and when it reaps them. */
+    libxl__sigchld_removehandler(ctx);
+
+    if (ctx->sigchld_selfpipe[0] >= 0) {
+        close(ctx->sigchld_selfpipe[0]);
+        close(ctx->sigchld_selfpipe[1]);
+    }
+
     pthread_mutex_destroy(&ctx->lock);
 
     GC_FREE;
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 8542e24..6e4c50c 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -620,6 +620,10 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
                                                                        \
         REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
                                                                        \
+        int selfpipe = libxl__fork_selfpipe_active(CTX);               \
+        if (selfpipe >= 0)                                             \
+            REQUIRE_FD(selfpipe, POLLIN, BODY);                        \
+                                                                       \
     }while(0)
 
 #define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
@@ -749,10 +753,11 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
                                int nfds, const struct pollfd *fds,
                                struct timeval now)
 {
+    /* May make callbacks into the appliction for child processes.
+     * ctx must be locked exactly once */
     EGC_GC;
     libxl__ev_fd *efd;
 
-
     LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
         if (!efd->events)
             continue;
@@ -763,11 +768,16 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
     }
 
     if (afterpoll_check_fd(poller,fds,nfds, poller->wakeup_pipe[0],POLLIN)) {
-        char buf[256];
-        int r = read(poller->wakeup_pipe[0], buf, sizeof(buf));
-        if (r < 0)
-            if (errno != EINTR && errno != EWOULDBLOCK)
-                LIBXL__EVENT_DISASTER(egc, "read wakeup", errno, 0);
+        int e = libxl__self_pipe_eatall(poller->wakeup_pipe[0]);
+        if (e) LIBXL__EVENT_DISASTER(egc, "read wakeup", e, 0);
+    }
+
+    int selfpipe = libxl__fork_selfpipe_active(CTX);
+    if (selfpipe >= 0 &&
+        afterpoll_check_fd(poller,fds,nfds, selfpipe, POLLIN)) {
+        int e = libxl__self_pipe_eatall(selfpipe);
+        if (e) LIBXL__EVENT_DISASTER(egc, "read sigchld pipe", e, 0);
+        libxl__fork_selfpipe_woken(egc);
     }
 
     for (;;) {
@@ -1063,16 +1073,36 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p)
 
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p)
 {
+    int e = libxl__self_pipe_wakeup(p->wakeup_pipe[1]);
+    if (e) LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", e, 0);
+}
+
+int libxl__self_pipe_wakeup(int fd)
+{
     static const char buf[1] = "";
 
     for (;;) {
-        int r = write(p->wakeup_pipe[1], buf, 1);
-        if (r==1) return;
+        int r = write(fd, buf, 1);
+        if (r==1) return 0;
         assert(r==-1);
         if (errno == EINTR) continue;
-        if (errno == EWOULDBLOCK) return;
-        LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", errno, 0);
-        return;
+        if (errno == EWOULDBLOCK) return 0;
+        assert(errno);
+        return errno;
+    }
+}
+
+int libxl__self_pipe_eatall(int fd)
+{
+    char buf[256];
+    for (;;) {
+        int r = read(fd, buf, sizeof(buf));
+        if (r >= 0) return 0;
+        assert(r == -1);
+        if (errno == EINTR) continue;
+        if (errno == EWOULDBLOCK) return 0;
+        assert(errno);
+        return errno;
     }
 }
 
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index 74a0402..869431d 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -161,11 +161,6 @@ void libxl_event_register_callbacks(libxl_ctx *ctx,
  * After libxl_ctx_free, all corresponding evgen handles become
  * invalid and must no longer be passed to evdisable.
  *
- * Events enabled with evenable prior to a fork and libxl_ctx_postfork
- * are no longer generated after the fork/postfork; however the evgen
- * structures are still valid and must be passed to evdisable if the
- * memory they use should not be leaked.
- *
  * Applications should ensure that they eventually retrieve every
  * event using libxl_event_check or libxl_event_wait, since events
  * which occur but are not retreived by the application will be queued
@@ -370,6 +365,142 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
 void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 
 
+/*======================================================================*/
+
+/*
+ * Subprocess handling.
+ *
+ * Unfortunately the POSIX interface makes this very awkward.
+ *
+ * There are two possible arrangements for collecting statuses from
+ * wait/waitpid.
+ *
+ * For naive programs:
+ *
+ *     libxl will keep a SIGCHLD handler installed whenever it has an
+ *     active (unreaped) child.  It will reap all children with
+ *     wait(); any children it does not recognise will be passed to
+ *     the application via an optional callback (and will result in
+ *     logged warnings if no callback is provided or the callback
+ *     denies responsibility for the child).
+ *
+ *     libxl may have children whenever:
+ *
+ *       - libxl is performing an operation which can be made
+ *         asynchronous; ie one taking a libxl_asyncop_how, even
+ *         if NULL is passed indicating that the operation is
+ *         synchronous; or
+ *
+ *       - events of any kind are being generated, as requested
+ *         by libxl_evenable_....
+ *
+ *     A multithreaded application which is naive in this sense may
+ *     block SIGCHLD on some of its threads, but there must be at
+ *     least one thread that has SIGCHLD unblocked.  libxl will not
+ *     modify the blocking flag for SIGCHLD (except that it may create
+ *     internal service threads with all signals blocked).
+ *
+ *     A naive program must only have at any one time only
+ *     one libxl context which might have children.
+ *
+ * For programs which run their own children alongside libxl's:
+ *
+ *     The application must install a SIGCHLD handler and reap (at
+ *     least) all of libxl's children and pass their exit status
+ *     to libxl by calling libxl_childproc_exited.
+ *
+ *     An application which does this must call
+ *     libxl_childproc_setmode.
+ * 
+ * An application which fails to call setmode, or which passes 0 for
+ * hooks, while it uses any libxl operation which might
+ * create or use child processes (see above):
+ *   - Must not have any child processes running.
+ *   - Must not install a SIGCHLD handler.
+ *   - Must not reap any children.
+ */
+
+
+typedef enum {
+    /* libxl owns SIGCHLD whenever it has a child. */
+    libxl_sigchld_owner_libxl,
+
+    /* Application promises to call libxl_childproc_exited but NOT
+     * from within a signal handler.  libxl will not itself arrange to
+     * (un)block or catch SIGCHLD. */
+    libxl_sigchld_owner_mainloop,
+
+    /* libxl owns SIGCHLD all the time, and the application is
+     * relying on libxl's event loop for reaping its own children. */
+    libxl_sigchld_owner_libxl_always,
+} libxl_sigchld_owner;
+
+typedef struct {
+    libxl_sigchld_owner chldowner;
+
+    /* All of these are optional: */
+
+    /* Called by libxl instead of fork.  Should behave exactly like
+     * fork, including setting errno etc.  May NOT reenter into libxl.
+     * Application may use this to discover pids of libxl's children,
+     * for example.
+     */
+    pid_t (*fork_replacement)(void *user);
+
+    /* With libxl_sigchld_owner_libxl, called by libxl when it has
+     * reaped a pid.  (Not permitted with _owner_mainloop.)
+     *
+     * Should return 0 if the child was recognised by the application
+     * (or if the application does not keep those kind of records),
+     * ERROR_NOT_READY if the application knows that the child is not
+     * the application's; if it returns another error code it is a
+     * disaster as described for libxl_event_register_callbacks.
+     * (libxl will report unexpected children to its error log.)
+     *
+     * If not supplied, the application is assumed not to start
+     * any children of its own.
+     *
+     * This function is NOT called from within the signal handler.
+     * Rather it will be called from inside a libxl's event handling
+     * code and thus only when libxl is running, for example from
+     * within libxl_event_wait.  (libxl uses the self-pipe trick
+     * to implement this.)
+     *
+     * childproc_exited_callback may call back into libxl, but it
+     * is best to avoid making long-running libxl calls as that might
+     * stall the calling event loop while the nested operation
+     * completes.
+     */
+    int (*reaped_callback)(pid_t, int status, void *user);
+} libxl_childproc_hooks;
+
+/* hooks may be 0 in which is equivalent to &{ libxl_sigchld_owner_libxl, 0, 0 }
+ *
+ * May not be called when libxl might have any child processes, or the
+ * behaviour is undefined.  So it is best to call this at
+ * initialisation.
+ */
+void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
+                             void *user);
+
+/*
+ * This function is for an application which owns SIGCHLD and which
+ * therefore reaps all of the process's children.
+ *
+ * May be called only by an application which has called setmode with
+ * chldowner == libxl_sigchld_owner_mainloop.  If pid was a process started
+ * by this instance of libxl, returns 0 after doing whatever
+ * processing is appropriate.  Otherwise silently returns
+ * ERROR_NOT_READY.  No other error returns are possible.
+ *
+ * May NOT be called from within a signal handler which might
+ * interrupt any libxl operation.  The application will almost
+ * certainly need to use the self-pipe trick (or a working pselect or
+ * ppoll) to implement this.
+ */
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status);
+
+
 /*
  * An application which initialises a libxl_ctx in a parent process
  * and then forks a child which does not quickly exec, must
@@ -379,6 +510,7 @@ void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 void libxl_postfork_child_noexec(libxl_ctx *ctx);
 
 
+
 #endif
 
 /*
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
index 66d0ee0..07d9ccf 100644
--- a/tools/libxl/libxl_fork.c
+++ b/tools/libxl/libxl_fork.c
@@ -46,6 +46,12 @@ static int atfork_registered;
 static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
     LIBXL_LIST_HEAD_INITIALIZER(carefds);
 
+/* non-null iff installed, protected by no_forking */
+static libxl_ctx *sigchld_owner;
+static struct sigaction sigchld_saved_action;
+
+static void sigchld_removehandler_core(void);
+
 static void atfork_lock(void)
 {
     int r = pthread_mutex_lock(&no_forking);
@@ -129,6 +135,7 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
     int r;
 
     atfork_lock();
+
     LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
         if (cf->fd >= 0) {
             r = close(cf->fd);
@@ -138,6 +145,10 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
         free(cf);
     }
     LIBXL_LIST_INIT(&carefds);
+
+    if (sigchld_owner)
+        sigchld_removehandler_core();
+
     atfork_unlock();
 }
 
@@ -159,6 +170,237 @@ int libxl__carefd_fd(const libxl__carefd *cf)
 }
 
 /*
+ * Actual child process handling
+ */
+
+static void sigchld_handler(int signo)
+{
+    int e = libxl__self_pipe_wakeup(sigchld_owner->sigchld_selfpipe[1]);
+    assert(!e); /* errors are probably EBADF, very bad */
+}
+
+static void sigchld_removehandler_core(void)
+{
+    struct sigaction was;
+    int r;
+    
+    r = sigaction(SIGCHLD, &sigchld_saved_action, &was);
+    assert(!r);
+    assert(!(was.sa_flags & SA_SIGINFO));
+    assert(was.sa_handler == sigchld_handler);
+    sigchld_owner = 0;
+}
+
+void libxl__sigchld_removehandler(libxl_ctx *ctx) /* non-reentrant */
+{
+    atfork_lock();
+    if (sigchld_owner == ctx)
+        sigchld_removehandler_core();
+    atfork_unlock();
+}
+
+int libxl__sigchld_installhandler(libxl_ctx *ctx) /* non-reentrant */
+{
+    int r, rc;
+
+    if (ctx->sigchld_selfpipe[0] < 0) {
+        r = pipe(ctx->sigchld_selfpipe);
+        if (r) {
+            ctx->sigchld_selfpipe[0] = -1;
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+                             "failed to create sigchld pipe");
+            rc = ERROR_FAIL;
+            goto out;
+        }
+    }
+
+    atfork_lock();
+    if (sigchld_owner != ctx) {
+        struct sigaction ours;
+
+        assert(!sigchld_owner);
+        sigchld_owner = ctx;
+
+        memset(&ours,0,sizeof(ours));
+        ours.sa_handler = sigchld_handler;
+        sigemptyset(&ours.sa_mask);
+        ours.sa_flags = SA_NOCLDSTOP | SA_RESTART;
+        r = sigaction(SIGCHLD, &ours, &sigchld_saved_action);
+        assert(!r);
+
+        assert(((void)"application must negotiate with libxl about SIGCHLD",
+                !(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
+                (sigchld_saved_action.sa_handler == SIG_DFL ||
+                 sigchld_saved_action.sa_handler == SIG_IGN)));
+    }
+    atfork_unlock();
+
+    rc = 0;
+ out:
+    return rc;
+}
+
+static int chldmode_ours(libxl_ctx *ctx)
+{
+    return ctx->childproc_hooks->chldowner == libxl_sigchld_owner_libxl;
+}
+
+int libxl__fork_selfpipe_active(libxl_ctx *ctx)
+{
+    /* Returns the fd to read, or -1 */
+    if (!chldmode_ours(ctx))
+        return -1;
+
+    if (LIBXL_LIST_EMPTY(&ctx->children))
+        return -1;
+
+    return ctx->sigchld_selfpipe[0];
+}
+
+static void perhaps_removehandler(libxl_ctx *ctx)
+{
+    if (LIBXL_LIST_EMPTY(&ctx->children) &&
+        ctx->childproc_hooks->chldowner != libxl_sigchld_owner_libxl_always)
+        libxl__sigchld_removehandler(ctx);
+}
+
+static int childproc_reaped(libxl__egc *egc, pid_t pid, int status)
+{
+    EGC_GC;
+    libxl__ev_child *ch;
+
+    LIBXL_LIST_FOREACH(ch, &CTX->children, entry)
+        if (ch->pid == pid)
+            goto found;
+
+    /* not found */
+    return ERROR_NOT_READY;
+
+ found:
+    LIBXL_LIST_REMOVE(ch, entry);
+    ch->callback(egc, ch, status);
+
+    perhaps_removehandler(CTX);
+
+    return 0;
+}
+
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t pid, int status)
+{
+    EGC_INIT(ctx);
+    CTX_LOCK;
+    int rc = childproc_reaped(egc, pid, status);
+    CTX_UNLOCK;
+    EGC_FREE;
+    return rc;
+}
+
+void libxl__fork_selfpipe_woken(libxl__egc *egc) {
+    /* May make callbacks into the appliction for child processes.
+     * ctx must be locked EXACTLY ONCE */
+    EGC_GC;
+
+    while (chldmode_ours(CTX) /* in case the app changes the mode */) {
+        int status;
+        pid_t pid = waitpid(-1, &status, WNOHANG);
+
+        if (pid == 0) return;
+
+        if (pid == -1) {
+            if (errno == ECHILD) return;
+            if (errno == EINTR) continue;
+            LIBXL__EVENT_DISASTER(egc, "waitpid() failed", errno, 0);
+            return;
+        }
+
+        int rc = childproc_reaped(egc, pid, status);
+
+        if (rc) {
+            if (CTX->childproc_hooks->reaped_callback) {
+                CTX_UNLOCK;
+                CTX->childproc_hooks->reaped_callback
+                    (pid, status, CTX->childproc_user);
+                CTX_LOCK;
+            } else {
+                LIBXL__LOG(CTX, LIBXL__LOG_WARNING, "unknown child [%ld]"
+                           " exited with status=%d", (long)pid, status);
+            }
+        }
+    }
+}
+
+int libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *ch,
+                         libxl__ev_child_callback *death)
+{
+    CTX_LOCK;
+    int rc;
+
+    if (chldmode_ours(CTX)) {
+        rc = libxl__sigchld_installhandler(CTX);
+        if (rc) goto out;
+    }
+
+    pid_t pid =
+        CTX->childproc_hooks->fork_replacement
+        ? CTX->childproc_hooks->fork_replacement(CTX->childproc_user)
+        : fork();
+    if (pid == -1) {
+        LIBXL__LOG_ERRNO(CTX, LIBXL__LOG_ERROR, "fork failed");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    if (!pid) {
+        /* woohoo! */
+        return 0; /* Yes, CTX is left locked in the child. */
+    }
+
+    ch->pid = pid;
+    ch->callback = death;
+    LIBXL_LIST_INSERT_HEAD(&CTX->children, ch, entry);
+    rc = pid;
+
+ out:
+    perhaps_removehandler(CTX);
+    CTX_UNLOCK;
+    return rc;
+}
+
+void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
+                             void *user)
+{
+    GC_INIT(ctx);
+    CTX_LOCK;
+
+    assert(LIBXL_LIST_EMPTY(&CTX->children));
+
+    if (!hooks)
+        hooks = &libxl__childproc_default_hooks;
+
+    ctx->childproc_hooks = hooks;
+    ctx->childproc_user = user;
+
+    switch (ctx->childproc_hooks->chldowner) {
+    case libxl_sigchld_owner_mainloop:
+    case libxl_sigchld_owner_libxl:
+        libxl__sigchld_removehandler(ctx);
+        break;
+    case libxl_sigchld_owner_libxl_always:
+        libxl__sigchld_installhandler(ctx);
+        break;
+    default:
+        abort();
+    }
+
+    CTX_UNLOCK;
+    GC_FREE;
+}
+
+const libxl_childproc_hooks libxl__childproc_default_hooks = {
+    libxl_sigchld_owner_libxl, 0, 0
+};
+
+/*
  * Local variables:
  * mode: C
  * c-basic-offset: 4
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 088a417..f4328f0 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -179,6 +179,19 @@ typedef struct libxl__ev_watch_slot {
 libxl__ev_xswatch *libxl__watch_slot_contents(libxl__gc *gc, int slotnum);
 
 
+typedef struct libxl__ev_child libxl__ev_child;
+typedef void libxl__ev_child_callback(libxl__egc *egc, libxl__ev_child*,
+                                      int status);
+struct libxl__ev_child {
+    /* caller should include this in their own struct */
+    /* read-only for caller: */
+    pid_t pid; /* -1 means unused ("unregistered", ie Idle) */
+    libxl__ev_child_callback *callback;
+    /* remainder is private for libxl__ev_... */
+    LIBXL_LIST_ENTRY(struct libxl__ev_child) entry;
+};
+
+
 /*
  * evgen structures, which are the state we use for generating
  * events for the caller.
@@ -248,6 +261,8 @@ struct libxl__ctx {
     const libxl_event_hooks *event_hooks;
     void *event_hooks_user;
 
+    LIBXL_LIST_ENTRY(struct libxl__ctx);
+
     pthread_mutex_t lock; /* protects data structures hanging off the ctx */
       /* Always use libxl__ctx_lock and _unlock (or the convenience
        * macors CTX_LOCK and CTX_UNLOCK) to manipulate this.
@@ -288,10 +303,14 @@ struct libxl__ctx {
     
     LIBXL_LIST_HEAD(, libxl_evgen_disk_eject) disk_eject_evgens;
 
-    /* for callers who reap children willy-nilly; caller must only
-     * set this after libxl_init and before any other call - or
-     * may leave them untouched */
+    const libxl_childproc_hooks *childproc_hooks;
+    void *childproc_user;
+    int sigchld_selfpipe[2]; /* [0]==-1 means handler not installed */
+    LIBXL_LIST_HEAD(, libxl__ev_child) children;
+
+    /* This is obsolete and must be removed: */
     int (*waitpid_instead)(pid_t pid, int *status, int flags);
+
     libxl_version_info version_info;
 };
 
@@ -534,6 +553,33 @@ static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw)
 
 
 /*
+ * For making subprocesses.  This is the only permitted mechanism for
+ * code in libxl to do so.
+ *
+ * In the parent, returns the pid, filling in childw_out.
+ * In the child, returns 0.
+ * If it fails, returns a libxl error (all of which are -ve).
+ *
+ * The child should go on to exec (or exit) soon, and should not make
+ * any further libxl event calls in the meantime.
+ *
+ * The parent may signal the child but it must not reap it.  That will
+ * be done by the event machinery.  Either death or cldstop may be
+ * NULL in which case that kind of event is ignored.
+ *
+ * It is not possible to "deregister" the child death event source.
+ * It will generate exactly one event callback; until then the childw
+ * is Active and may not be reused.
+ */
+_hidden int libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *childw_out,
+                                 libxl__ev_child_callback *death);
+static inline void libxl__ev_child_init(libxl__ev_child *childw_out)
+                { childw_out->pid = -1; }
+static inline int libxl__ev_child_inuse(libxl__ev_child *childw_out)
+                { return childw_out->pid >= 0; }
+
+
+/*
  * Other event-handling support provided by the libxl event core to
  * the rest of libxl.
  */
@@ -587,6 +633,15 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
  * ctx must be locked. */
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
 
+/* Internal to fork and child reaping machinery */
+extern const libxl_childproc_hooks libxl__childproc_default_hooks;
+int libxl__sigchld_installhandler(libxl_ctx *ctx); /* non-reentrant;logs errs */
+void libxl__sigchld_removehandler(libxl_ctx *ctx); /* non-reentrant */
+int libxl__fork_selfpipe_active(libxl_ctx *ctx); /* returns read fd or -1 */
+void libxl__fork_selfpipe_woken(libxl__egc *egc);
+int libxl__self_pipe_wakeup(int fd); /* returns 0 or -1 setting errno */
+int libxl__self_pipe_eatall(int fd); /* returns 0 or -1 setting errno */
+
 
 int libxl__atfork_init(libxl_ctx *ctx);
 
-- 
1.7.2.5