[PATCH 0/6][v3][lxc] Link LXC with USERCR

[PATCH 0/6][v3][lxc] Link LXC with USERCR

[Sukadev Bhattiprolu]

Resending patches after fixing whitespace damage and the typo in
configure.ac. Add a section to patch-0 (below) to describe the
kernel build (commit-id, config tokens etc).

---
Following set of patches, when applied to LXC 0.6.5 and built with usercr,
enable lxc_checkpoint and lxc_restart of some simple containers.

TODO:
	- Determine if lxc_checkpoint needs a --container option (see
	  TODOs in src/lxc/checkpoint.c)
	- This patchset was tested using lxc-nonetns.conf. I ran into a problem
	  creating a bridge with lxc-veth.conf and lxc-macvlan.conf.  I have
	  not debugged the problem with VNC and lxc-macvlan.conf.
	- 'global_send_sigint' is still a global variable in USERCR. We need
	  to define a better interface to expose its functionality to callers
	  of app_restart().
	- Choose better names and API for USERCR :-)
	- Additional TODOs specific to checkpoint/restart are listed in their
	  specific patches.

Changelog[v3]:
	- Renamed --image option to --statefile
	- Added --with-libcr configuration option to specify the path to
	  usercr (see usage below).
	- lxc-checkpoint now implicitly freezes before and unfreezes after
	  checkpoint.
	- Implemented the --pause options to lxc-checkpoint and lxc-restart
	  and the --kill option to lxc-checkpoint.
	- Ported to ckpt-v20-dev (required adding CHECKPOINT_NONETNS flag to
	  app_checkpoint() to work around)

Changelog[v2]: 

	(Based on feedback from Oren Laadan, Serge Hallyn, Daniel Lezcano
	and Cedric Le Goater)

	- Rather than drop --directory option to lxc_checkpoint/lxc_restart
	  add a new option (--image).
	- Integrate lxc_checkpoint to work with USERCR
	- USERCR renamed usercr.h to "app-checkpoint.h"
	- USERCR does not create/install libcheckpoint.a and usercr.h for now.
	  So link directly with app-checkpoint.h, restart.o and checkpoint.o
	- USERCR renames the interfaces to app_checkpoint() and app_restart()
	  'struct app_checkpoint_args' 'struct app_restart_args'.
	  
USAGE:

1. Build C/R-enabled Linux kernel

	$ cd /root

	$ git-clone  git://www.linux-cr.org/pub/git/linux-cr.git linux-cr

	$ cd linux-cr

	$ git-checkout ckpt-v20-dev

		I tested with commit 3522c57a9ec6f08a129a78322318abcb4467db28 
		as HEAD.

	# Ensure following tokens are set in .config

	CONFIG_CHECKPOINT_SUPPORT=y
	CONFIG_SYSVIPC_CHECKPOINT=y
	CONFIG_CHECKPOINT=y
	CONFIG_CHECKPOINT_NETNS=y
	CONFIG_CHECKPOINT_DEBUG=y

	CONFIG_CGROUPS=y
	CONFIG_CGROUP_FREEZER=y
	CONFIG_NAMESPACES=y
	CONFIG_CGROUP_NS=y
	CONFIG_UTS_NS=y

	CONFIG_IPC_NS=y
	CONFIG_USER_NS=y
	CONFIG_PID_NS=y
	CONFIG_NET_NS=y

	# Build, install, reboot on new kernel

	# After every reboot, ensure '-o newinstance' mount option to /dev/pts
	  works (see Documentation/filesystems/devpts.txt for details). In
	  short, run following commands on each reboot:

		$ rm /dev/ptmx

		$ ln -s pts/ptmx /dev/ptmx

		$ chmod 666 /dev/pts/ptmx


1. Build USERCR

	$ cd /root

	$ git-clone git://git.ncl.cs.columbia.edu/pub/git/user-cr.git user-cr

	$ cd user-cr

	$ git-checkout ckpt-v20-dev

	  	Tested with commit e275f77e4a82d228c1df14dbeb691342e32cdac2
		as HEAD.
	
	# Apply following two patches:

	https://lists.linux-foundation.org/pipermail/containers/2010-March/024037.html
	https://lists.linux-foundation.org/pipermail/containers/2010-March/024038.html

	$ KERNELSRC=/root/linux-cr make 

		Build USERCR by pointing to corresponding kernel-source.
		This should create restart.o and checkpoint.o needed by LXC.

		I did not need to, but you may need to compile checkpoint.o
		and restart.o with -fPIC compiler option.

	$ make install

2. Build/install LXC

	$ cd /root

	$ git-clone git://lxc.git.sourceforge.net/gitroot/lxc/lxc lxc.git

	$ cd lxc.git

	# Apply attached patches to LXC (I tested with these patches applied
	to commit 9ea8066aa67b808f71f46e346bd7a215e2a355f3)

	$ autogen.sh

	$ ./configure --with-libcr=/root/user-cr

		This will fail if /root/user-cr does not container checkpoint.o,
		restart.o and app-checkpoint.h files 
	$ make

	$ make install

3. Checkpoint/restart a simple LXC container

	$ lxc-execute --name foo --rcfile lxc-no-netns.conf -- /bin/sleep 1000

	$ lxc-checkpoint --name foo --statefile /root/lxc-foo.ckpt

	$ lxc-stop --name foo

	$ lxc-restart --name foo --statefile /root/lxc-foo.ckpt

4. Checkpoint/restart other LXC containers such as:

	- a file-io session (see run-fileio1 in cr-tests[1])

	- process-tree (see run-ptree1 in cr-tests[1])

	- A vi session inside a VNC Server using "twm". i.e

		$ cat /root/.vnc/xstartup
		#!/bin/sh

		xsetroot -solid grey
		vncconfig -iconic &
		xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
		twm &

		$ lxc-execute --name foo --rcfile lxc-no-netns.conf -- \
			/usr/bin/vncserver :1

		$ vncviewer :1

			# Open a vi session

		$ lxc-checkpoint --name foo  --statefile /root/vnc.ckpt

		$ lxc-stop --name foo

		$ lxc-restart --pause --name foo --statefile /root/vnc.ckpt

			# Leaves the server frozen due to --pause

		$ lxc-unfreeze --name foo

		$ vncviewer :1

			# Should bring up the old VNC session with vi window

[1]: cr-tests:	git://git.sr71.net/~hallyn/cr_tests.git

Signed-off-by: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>

[PATCH 1/6][v3][lxc] Add --with-libcr configure option

[Sukadev Bhattiprolu]

From: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
Date: Wed, 24 Mar 2010 17:26:44 -0700
Subject: [PATCH 1/6][v3][lxc] Add --with-libcr configure option

Add a configure option, --with-libcr=dir which would allow linking
with external (i.e USERCR) implementation  of checkpoint/restart.

For now, USERCR "publishes" a app-checkpoint.h, checkpoint.o and
restart.o files which implement the functions app_checkpoint() and
app_restart().

Usage:
	$ ./autogen.sh

	$ ./configure --help |grep libcr
	--with-libcr=dir     use the Checkpoint/Restart implementation in 'dir'

	$ ls /home/guest/user-cr/
	app-checkpoint.h    checkpoint.o    restart.o

	$ ./configure --with-libcr=/home/guest/user-cr

TODO:
	If names of interfaces in USERCR change, we may want to rename
	the config option too ?

	LIBCR_CFLAGS are only needed for src/lxc/{checkpoint.c,restart.c}
	but not sure if there is an easy way to define autoconf CFLAGS
	just for those two files.

Changelog[v2]:
	- Rename --with-usercr to --with-libcr
	- Add libeclone.a to the LIBCR_OBJS variable since functions in
	  libeclone.a will be used by checkpoint() and restart() functions.
	- Add -I${with_libcr}/include to LIBCR_CFLAGS to pick up
	  checkpoint_hdr.h, checkpoint.h etc.

Signed-off-by: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---
 configure.ac        |   19 +++++++++++++++++++
 src/lxc/Makefile.am |   10 +++++++++-
 2 files changed, 28 insertions(+), 1 deletions(-)

diff --git a/configure.ac b/configure.ac
index f82e7df..b5a4ff7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -12,6 +12,25 @@ AM_PROG_CC_C_O
 AC_GNU_SOURCE
 AC_CHECK_PROG(SETCAP, setcap, yes, no, $PATH$PATH_SEPARATOR/sbin)
 
+AC_ARG_WITH(libcr, [AS_HELP_STRING([--with-libcr=dir], \
+           [use the Checkpoint/Restart implementation in 'dir'])], [], \
+	   [with_libcr=no])
+
+if test "x$with_libcr" != "xno"; then
+       AS_AC_EXPAND(LIBCR_OBJS, "${with_libcr}/checkpoint.o ${with_libcr}/restart.o ${with_libcr}/libeclone.a")
+       AS_AC_EXPAND(LIBCR_CFLAGS, "-DLIBCR -I${with_libcr} -I${with_libcr}/include")
+
+       AC_CHECK_FILE([$with_libcr/app-checkpoint.h], [], \
+               AC_MSG_ERROR([--with-libcr specified directory $with_libcr but $with_libcr/app-checkpoint.h was not found]))
+
+       AC_CHECK_FILE([${with_libcr}/checkpoint.o], [], \
+               AC_MSG_ERROR([--with-libcr specified directory $with_libcr but ${with_libcr}/checkpoint.o was not found]))
+
+       AC_CHECK_FILE([${with_libcr}/restart.o], [], \
+               AC_MSG_ERROR([--with-libcr specified directory $with_libcr but ${with_libcr}/restart.o was not found]))
+fi
+
+
 AC_ARG_ENABLE([doc],
 	[AC_HELP_STRING([--enable-doc], [make mans (require docbook2man installed) [default=auto]])],
 	[], [enable_doc=auto])
diff --git a/src/lxc/Makefile.am b/src/lxc/Makefile.am
index 890f706..699c355 100644
--- a/src/lxc/Makefile.am
+++ b/src/lxc/Makefile.am
@@ -46,12 +46,20 @@ liblxc_so_SOURCES = \
 	mainloop.c mainloop.h \
 	af_unix.c af_unix.h
 
-AM_CFLAGS=-I$(top_srcdir)/src
+# We only need $(LIBCR_CFLAGS) for lxc_checkpoint and lxc_restart files
+# but for now, just set it for all.
+AM_CFLAGS=-I$(top_srcdir)/src $(LIBCR_CFLAGS)
 
 liblxc_so_CFLAGS = -fPIC -DPIC $(AM_CFLAGS)
 
+# TODO: Adding $(LIBCR_OBJS) here ensures we don't have undefined references
+# 	when building liblxc.so, but this has the side-effect of putting the
+# 	app_checkpoint/restart functions in liblxc.so. Or alternatively,
+# 	we could remove src/lxc/{checkpoint.o,restart.o} from liblxc.so
+# 	and link lxc-checkpoint/lxc-restart with them directly.
 liblxc_so_LDFLAGS = \
 	-shared \
+	$(LIBCR_OBJS) \
 	-Wl,-soname,liblxc.so.$(firstword $(subst ., ,$(VERSION)))
 
 liblxc_so_LDADD = -lutil
-- 
1.6.0.4

[PATCH 2/6][v3][lxc] lxc_restart: Add --statefile option

[Sukadev Bhattiprolu]

From: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
Date: Sat, 27 Mar 2010 00:08:17 -0700
Subject: [PATCH 2/6][v3][lxc] lxc_restart: Add --statefile option

The existing --directory option to lxc_restart expects the checkpoint state
to be a directory. USERCR however uses a single regular file to store the
checkpoint image. So add a --statefile option to enable checkpointing and
restarting applications using USERCR.

Depending on how the application was checkpointed, users should specify
either --statefile=STATEFILE or the --directory=STATEFILE option (but not
both).

Signed-off-by: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---
 src/lxc/lxc_restart.c |   30 +++++++++++++++++++++++-------
 1 files changed, 23 insertions(+), 7 deletions(-)

diff --git a/src/lxc/lxc_restart.c b/src/lxc/lxc_restart.c
index 7db1d85..4be6f92 100644
--- a/src/lxc/lxc_restart.c
+++ b/src/lxc/lxc_restart.c
@@ -38,11 +38,19 @@
 lxc_log_define(lxc_restart_ui, lxc_restart);
 
 static struct lxc_list defines;
+static char *statedir;
 
 static int my_checker(const struct lxc_arguments* args)
 {
-	if (!args->statefile) {
-		lxc_error(args, "no statefile specified");
+	int d, f;
+
+	/* make them boolean */
+	d = !!(statedir);
+	f = !!(args->statefile);
+
+	if (!(d ^ f)) {
+		lxc_error(args, "Must specify exactly one of --directory "
+				"and --statefile options");
 		return -1;
 	}
 
@@ -52,8 +60,9 @@ static int my_checker(const struct lxc_arguments* args)
 static int my_parser(struct lxc_arguments* args, int c, char* arg)
 {
 	switch (c) {
-	case 'd': args->statefile = arg; break;
+	case 'd': statedir = arg; break;
 	case 'f': args->rcfile = arg; break;
+	case 'S': args->statefile = arg; break;
 	case 'p': args->flags = LXC_FLAG_PAUSE; break;
 	case 's': return lxc_config_define_add(&defines, arg);
 	}
@@ -66,21 +75,24 @@ static const struct option my_longopts[] = {
 	{"rcfile", required_argument, 0, 'f'},
 	{"pause", no_argument, 0, 'p'},
 	{"define", required_argument, 0, 's'},
+	{"statefile", required_argument, 0, 'S'},
 	LXC_COMMON_OPTIONS
 };
 
 static struct lxc_arguments my_args = {
 	.progname = "lxc-restart",
 	.help     = "\
---name=NAME --directory STATEFILE\n\
+--name=NAME --directory STATEFILE (deprecated)\n\
+\tlxc_restart --name=NAME --statefile=STATEFILE\n\
 \n\
 lxc-restart restarts from STATEFILE the NAME container\n\
 \n\
 Options :\n\
   -n, --name=NAME      NAME for name of the container\n\
   -p, --pause          do not release the container after the restart\n\
-  -d, --directory=STATEFILE for name of statefile\n\
+  -d, --directory=STATEFILE for name of statefile (legacy mode, deprecated)\n\
   -f, --rcfile=FILE Load configuration file FILE\n\
+  -i, --statefile=STATEFILE Load the application state from STATEFILE (libcr mode)\n\
   -s, --define KEY=VAL Assign VAL to configuration variable KEY\n",
 	.options  = my_longopts,
 	.parser   = my_parser,
@@ -90,6 +102,7 @@ Options :\n\
 int main(int argc, char *argv[])
 {
 	char *rcfile = NULL;
+	const char *statefile;
 	struct lxc_conf *conf;
 
 	lxc_list_init(&defines);
@@ -131,6 +144,9 @@ int main(int argc, char *argv[])
 	if (lxc_config_define_load(&defines, conf))
 		return -1;
 
-	return lxc_restart(my_args.name, my_args.statefile, conf,
-			   my_args.flags);
+	statefile = my_args.statefile;
+	if (statedir)
+		statefile = statedir;
+
+	return lxc_restart(my_args.name, statefile, conf, my_args.flags);
 }
-- 
1.6.0.4

[PATCH 3/6][v3][lxc] lxc_checkpoint: Add --statefile option

[Sukadev Bhattiprolu]

From: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
Date: Wed, 10 Mar 2010 22:24:17 -0800
Subject: [PATCH 3/6][v3][lxc] lxc_checkpoint: Add --statefile option

The existing --directory option to lxc_checkpoint expects to save the
checkpoint state in a directory. USERCR however uses a single regular
file to store the checkpoint image. So add a --statefile option to enable
checkpointing and restarting applications using USERCR.

Users should specify either --statefile or --directory option (but not both)
to select the file/directory where the application state will be stored.

Signed-off-by: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---
 src/lxc/lxc_checkpoint.c |   36 +++++++++++++++++++++++++++---------
 1 files changed, 27 insertions(+), 9 deletions(-)

diff --git a/src/lxc/lxc_checkpoint.c b/src/lxc/lxc_checkpoint.c
index a8c74a9..b0cd634 100644
--- a/src/lxc/lxc_checkpoint.c
+++ b/src/lxc/lxc_checkpoint.c
@@ -37,10 +37,19 @@
 
 lxc_log_define(lxc_checkpoint_ui, lxc_checkpoint);
 
+static char *statedir;
+
 static int my_checker(const struct lxc_arguments* args)
 {
-	if (!args->statefile) {
-		lxc_error(args, "no statefile specified");
+	int d, f;
+
+	/* make them boolean */
+	d = !!(statedir);
+	f = !!(args->statefile);
+
+	if (!(d ^ f)) {
+		lxc_error(args, "Must specify exactly one of --directory "
+				"and --statefile options");
 		return -1;
 	}
 
@@ -52,7 +61,8 @@ static int my_parser(struct lxc_arguments* args, int c, char* arg)
 	switch (c) {
 	case 'k': args->flags = LXC_FLAG_HALT; break;
 	case 'p': args->flags = LXC_FLAG_PAUSE; break;
-	case 'd': args->statefile = arg; break;
+	case 'd': statedir = arg; break;
+	case 'S': args->statefile = arg; break;
 	}
 	return 0;
 }
@@ -61,13 +71,15 @@ static const struct option my_longopts[] = {
 	{"kill", no_argument, 0, 'k'},
 	{"pause", no_argument, 0, 'p'},
 	{"directory", required_argument, 0, 'd'},
+	{"statefile", required_argument, 0, 'S'},
 	LXC_COMMON_OPTIONS
 };
 
 static struct lxc_arguments my_args = {
 	.progname = "lxc-checkpoint",
 	.help     = "\
---name=NAME --directory STATEFILE\n\
+--name=NAME --directory STATEFILE (deprecated)\n\
+\tlxc_checkpoint --name=NAME --statefile=STATEFILE\n\
 \n\
 lxc-checkpoint checkpoints in STATEFILE the NAME container\n\
 \n\
@@ -75,7 +87,8 @@ Options :\n\
   -n, --name=NAME      NAME for name of the container\n\
   -k, --kill           stop the container after checkpoint\n\
   -p, --pause          don't unfreeze the container after the checkpoint\n\
-  -d, --directory=STATEFILE where to store the statefile\n",
+  -d, --directory=STATEFILE where to store the statefile (deprecated)\n\
+  -i, --statefile=STATEFILE where to store the checkpoint-image (LIBCR mode)\n",
 
 	.options  = my_longopts,
 	.parser   = my_parser,
@@ -97,6 +110,7 @@ static int create_statefile(const char *dir)
 int main(int argc, char *argv[])
 {
 	int ret;
+	const char *statefile;
 
 	ret = lxc_arguments_parse(&my_args, argc, argv);
 	if (ret)
@@ -107,11 +121,15 @@ int main(int argc, char *argv[])
 	if (ret)
 		return ret;
 
-	ret = create_statefile(my_args.statefile);
-	if (ret)
-		return ret;
+	statefile = my_args.statefile;
+	if (statedir) {
+		statefile = statedir;
+		ret = create_statefile(statefile);
+		if (ret)
+			return ret;
+	}
 
-	ret = lxc_checkpoint(my_args.name, my_args.statefile, my_args.flags);
+	ret = lxc_checkpoint(my_args.name, statefile, my_args.flags);
 	if (ret)
 		ERROR("failed to checkpoint '%s'", my_args.name);
 	else
-- 
1.6.0.4

[PATCH 4/6][v3][lxc] Move get_init_pid() into checkpoint.c

[Sukadev Bhattiprolu]

From: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
Date: Mon, 29 Mar 2010 23:53:55 -0700
Subject: [PATCH 4/6][v3][lxc] Move get_init_pid() into checkpoint.c

lxc_attach.c is currently not included in liblxc.so. In  afollowon
patch, checkpoint() function needs to also use the get_init_pid()
interface. So move the defintions into checkpoint.c - which would
then be accessible to both lxc_attach and lxc-checkpoint.

Signed-off-by: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---
 src/lxc/checkpoint.c |   29 +++++++++++++++++++++++++++++
 src/lxc/lxc.h        |    6 ++++++
 src/lxc/lxc_attach.c |   29 +----------------------------
 3 files changed, 36 insertions(+), 28 deletions(-)

diff --git a/src/lxc/checkpoint.c b/src/lxc/checkpoint.c
index 7e8a93e..4e75cb6 100644
--- a/src/lxc/checkpoint.c
+++ b/src/lxc/checkpoint.c
@@ -22,9 +22,38 @@
  */
 #include <lxc/lxc.h>
 #include <lxc/log.h>
+#include <lxc/commands.h>
 
 lxc_log_define(lxc_checkpoint, lxc);
 
+pid_t get_init_pid(const char *name)
+{
+	struct lxc_command command = {
+		.request = { .type = LXC_COMMAND_PID },
+	};
+
+	int ret, stopped = 0;
+
+	ret = lxc_command(name, &command, &stopped);
+	if (ret < 0 && stopped) {
+		INFO("'%s' is already stopped", name);
+		return 0;
+	}
+
+	if (ret < 0) {
+		ERROR("failed to send command");
+		return -1;
+	}
+
+	if (command.answer.ret) {
+		ERROR("failed to retrieve the init pid: %s",
+		      strerror(-command.answer.ret));
+		return -1;
+	}
+
+	return command.answer.pid;
+}
+
 int lxc_checkpoint(const char *name, const char *statefile, int flags)
 {
 	return 0;
diff --git a/src/lxc/lxc.h b/src/lxc/lxc.h
index b0b9f4e..bd87bdb 100644
--- a/src/lxc/lxc.h
+++ b/src/lxc/lxc.h
@@ -27,6 +27,7 @@
 extern "C" {
 #endif
 
+#include <unistd.h>
 #include <stddef.h>
 #include <lxc/state.h>
 
@@ -56,6 +57,11 @@ extern int lxc_start(const char *name, char *const argv[], struct lxc_conf *);
 extern int lxc_stop(const char *name);
 
 /*
+ * Get the pid of the root application process tree in parent-pid namespace
+ */
+extern pid_t get_init_pid(const char *name);
+
+/*
  * Open the monitoring mechanism for a specific container
  * The function will return an fd corresponding to the events
  * Returns a file descriptor on success, < 0 otherwise
diff --git a/src/lxc/lxc_attach.c b/src/lxc/lxc_attach.c
index a012c2c..3d2cdd5 100644
--- a/src/lxc/lxc_attach.c
+++ b/src/lxc/lxc_attach.c
@@ -28,6 +28,7 @@
 #include "commands.h"
 #include "arguments.h"
 #include "namespace.h"
+#include "lxc.h"
 #include "log.h"
 
 lxc_log_define(lxc_attach_ui, lxc);
@@ -50,34 +51,6 @@ Options :\n\
 	.checker  = NULL,
 };
 
-pid_t get_init_pid(const char *name)
-{
-	struct lxc_command command = {
-		.request = { .type = LXC_COMMAND_PID },
-	};
-
-	int ret, stopped = 0;
-
-	ret = lxc_command(name, &command, &stopped);
-	if (ret < 0 && stopped) {
-		INFO("'%s' is already stopped", name);
-		return 0;
-	}
-
-	if (ret < 0) {
-		ERROR("failed to send command");
-		return -1;
-	}
-
-	if (command.answer.ret) {
-		ERROR("failed to retrieve the init pid: %s",
-		      strerror(-command.answer.ret));
-		return -1;
-	}
-
-	return command.answer.pid;
-}
-
 int main(int argc, char *argv[], char *envp[])
 {
 	int ret;
-- 
1.6.0.4

[PATCH 5/6][v3][lxc] Hook up lxc_restart() with app_restart()

[Sukadev Bhattiprolu]

From: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
Date: Thu, 11 Mar 2010 20:47:56 -0800
Subject: [PATCH 5/6][v3][lxc] Hook up lxc_restart() with app_restart()

Have lxc_restart() call app_restart() implemented in the 'restart.o' from
USER-CR git tree.

Changelog[v3]:
	- (Daniel Lezcano) Remove unnecessary check for 'statefile' before
	  opening it
	- [Daniel Lezcano] Rebase to more recent version and use
	  lxc_check_inherited() instead of lxc_close_inherited_fd().
	- Have lxc_restart unfreeze the container if the --pause option
	  was not specified.
	- Use -D LIBCR to fix compile error when --with-libcr config is
	  not specified. Return ENOSYS if built with LIBCR undefined.
	- Implement the --pause option to set the RESTART_FROZEN flag.

Changelog[v2]:
	- Link with restart.o from usercr rather than libcheckpoint.a
	- rename 'struct restart_args' to 'struct app_restart_args'
	- Initialize the new field app_restart_args->uerrfd
	- (Oren Laadan)Remove ->send_sigint field from 'struct app_restart_args'

Signed-off-by: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---
 src/lxc/restart.c |  129 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 128 insertions(+), 1 deletions(-)

diff --git a/src/lxc/restart.c b/src/lxc/restart.c
index 467489e..c9ff565 100644
--- a/src/lxc/restart.c
+++ b/src/lxc/restart.c
@@ -22,11 +22,138 @@
  */
 #include <lxc/lxc.h>
 #include <lxc/log.h>
+#include <lxc/start.h>
+#include <lxc/namespace.h>
+#include <lxc/cgroup.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <signal.h>
+#include <sys/prctl.h>
+#include <sys/wait.h>
+#include "error.h"
+#ifdef LIBCR
+#include <app-checkpoint.h>
+#endif
 
 lxc_log_define(lxc_restart, lxc);
 
+#ifdef LIBCR
+
+struct lxc_restart_arg {
+	const char *name;
+	const char *statefile;
+	char *const argv;
+	struct lxc_handler *handler;
+	int lxc_flags;
+};
+
+static int do_restart(struct lxc_restart_arg *lxcarg)
+{
+	int pid;
+	int lxc_flags = lxcarg->lxc_flags;
+	struct lxc_handler *handler = lxcarg->handler;
+	const char *statefile = lxcarg->statefile;
+	struct app_restart_args restart_args;
+
+	if (sigprocmask(SIG_SETMASK, &handler->oldmask, NULL)) {
+		SYSERROR("failed to set sigprocmask");
+		return -1;
+	}
+
+	if (prctl(PR_SET_PDEATHSIG, SIGKILL, 0, 0, 0)) {
+		SYSERROR("failed to set pdeath signal");
+		return -1;
+	}
+
+	memset(&restart_args, 0, sizeof(restart_args));
+
+	restart_args.infd = open(statefile, O_RDONLY, 0);
+	if (restart_args.infd < 0) {
+		SYSERROR("Failed to open statefile %s\n", statefile);
+		return -1;
+	}
+
+	restart_args.pids = 1;
+	restart_args.pidns = 1;
+	restart_args.mnt_pty = 1;
+	restart_args.mntns = 1;
+	restart_args.klogfd = -1;
+	restart_args.ulogfd = lxc_log_fd;
+	restart_args.uerrfd = fileno(stderr);
+	restart_args.debug = 1;
+	restart_args.wait = 0;
+
+	if (lxc_flags & LXC_FLAG_PAUSE)
+		restart_args.keep_frozen = 1;
+
+	pid = app_restart(&restart_args);
+
+	return pid;
+}
+
 int lxc_restart(const char *name, const char *statefile, struct lxc_conf *conf,
-		int flags)
+		int lxc_flags)
 {
+	int err;
+	int status;
+	struct lxc_handler *handler;
+	struct lxc_restart_arg lxcarg = {
+		.name = name,
+		.statefile = statefile,
+		.lxc_flags = lxc_flags,
+		.handler = NULL,
+	};
+
+	if (lxc_check_inherited())
+		return -1;
+
+	handler = lxc_init(name, conf);
+	if (!handler) {
+		ERROR("failed to initialize the container");
+		return -1;
+	}
+
+	lxcarg.handler = handler;
+	handler->pid = do_restart(&lxcarg);
+
+	INFO("do_restart(): returns pid %d\n", handler->pid);
+	lxc_rename_nsgroup(name, handler);
+
+	err = lxc_poll(name, handler);
+	if (err) {
+		ERROR("mainloop exited with an error");
+		goto out_abort;
+	}
+
+	while (waitpid(handler->pid, &status, 0) < 0 && errno == EINTR)
+		continue;
+
+	if (!(lxc_flags & LXC_FLAG_PAUSE)) {
+		err = lxc_unfreeze(name);
+		if (err) {
+			ERROR("lxc_restart(): Unable to unfreeze\n");
+			goto out_fini;
+		}
+	}
+
+	err =  lxc_error_set_and_log(handler->pid, status);
+
+out_fini:
+	lxc_fini(name, handler);
+	return err;
+
+out_abort:
+	lxc_abort(name, handler);
+	goto out_fini;
+
 	return 0;
 }
+#else
+int lxc_restart(const char *name, const char *statefile, struct lxc_conf *conf,
+		int lxc_flags)
+{
+	ERROR("'restart' function not configured");
+	ERROR("Try --with-libcr option to 'configure' script");
+	return -1;
+}
+#endif
-- 
1.6.0.4

[PATCH 6/6][v3][lxc] Hook up lxc_checkpoint() with app_checkpoint()

[Sukadev Bhattiprolu]

From: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
Date: Thu, 11 Mar 2010 21:32:38 -0800
Subject: [PATCH 6/6][v3][lxc] Hook up lxc_checkpoint() with app_checkpoint()

Have lxc_checkpoint() call app_checkpoint() implemented in checkpoint.o
in the USER-CR git tree

TODO:
	- Map lxc_flags to flags in sys_checkpoint()
	- Initialize app_checkpoint_args.debug and other fields based on
	  command line options to lxc_checkpoint rather than hard-coding
	  them

Changelog:[v2]:
	- Drop find_cinit_pid() and use get_init_pid() from a recent checkin
	- Implement --pause and --kill options to lxc-checkpoint
	- Use -D LIBCR to fix compile error when --with-libcr config option
	  is not specified. Return ENOSYS when LIBCR is undefined.
	- Add CHECKPOINT_NONETNS to flags arg to app_checkpoint() which
	  is needed to checkpoint an application that is not in a private
	  netns (new in ckpt-v20-dev).

Signed-off-by: Sukadev Bhattiprolu <sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
---
 src/lxc/checkpoint.c |   88 ++++++++++++++++++++++++++++++++++++++++++++++++-
 src/lxc/state.c      |    1 -
 2 files changed, 86 insertions(+), 3 deletions(-)

diff --git a/src/lxc/checkpoint.c b/src/lxc/checkpoint.c
index 4e75cb6..d861ce1 100644
--- a/src/lxc/checkpoint.c
+++ b/src/lxc/checkpoint.c
@@ -22,7 +22,17 @@
  */
 #include <lxc/lxc.h>
 #include <lxc/log.h>
-#include <lxc/commands.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <linux/checkpoint.h>
+
+#include "commands.h"
+#include "arguments.h"
+#include "namespace.h"
+#ifdef LIBCR
+#include "app-checkpoint.h"
+#endif
 
 lxc_log_define(lxc_checkpoint, lxc);
 
@@ -54,7 +64,81 @@ pid_t get_init_pid(const char *name)
 	return command.answer.pid;
 }
 
-int lxc_checkpoint(const char *name, const char *statefile, int flags)
+#ifdef LIBCR
+int lxc_checkpoint(const char *name, const char *statefile, int lxc_flags)
 {
+	int ret;
+	int pid;
+	int flags;
+	struct app_checkpoint_args crargs;
+
+	pid = get_init_pid(name);
+	if (pid < 0) {
+		ERROR("Unable to find cinit pid");
+		return -1;
+	}
+
+	ret = lxc_freeze(name);
+	if (ret < 0)
+		return ret;
+
+	memset(&crargs, 0, sizeof(crargs));
+
+	ret = open(statefile, O_CREAT|O_RDWR|O_EXCL, 0600);
+	if (ret < 0) {
+		ERROR("open(%s) failed, %s\n", statefile, strerror(errno));
+		return -1;
+	}
+
+	crargs.outfd = ret;
+	crargs.logfd = lxc_log_fd;
+	crargs.uerrfd = lxc_log_fd;
+	/*
+	 * TODO: Set this to 0 for now - otherwise we get an objhash leak
+	 * 	 due to mismatched references to current PTY which needs to
+	 * 	 be investigated.
+	 *
+	 * TODO: Map @lxc_flags to user-cr flags ?
+	 *
+	 * TODO: We can probably drop the ->container field since @flags
+	 * 	 can provide the same selection.
+	 *
+	 * TODO: Do we may need a --container option to lxc_checkpoint or
+	 * 	 assume that we always work with full containers ?
+	 */
+	crargs.container = 0;
+
+	/*
+	 * TODO: Set the CHECKPOINT_NONETNS unconditionally for now. Otherwise
+	 * 	 it would require running the application in a private netns.
+	 * 	 Implement a command-line option to allow user selection.
+	 */
+	flags = CHECKPOINT_SUBTREE|CHECKPOINT_NONETNS;
+
+	ret = app_checkpoint(pid, flags, &crargs);
+	if (ret < 0) {
+		ERROR("checkpoint of %s (pid %d) failed\n", name, pid);
+		return -1;
+	}
+
+	if (lxc_flags & LXC_FLAG_HALT) {
+		ret = lxc_stop(name);
+		if (ret < 0)
+			return ret;
+
+		return lxc_unfreeze(name);
+	}
+
+	if (!(lxc_flags & LXC_FLAG_PAUSE))
+		return lxc_unfreeze(name);
+
 	return 0;
 }
+#else
+int lxc_checkpoint(const char *name, const char *statefile, int lxc_flags)
+{
+	ERROR("'checkpoint' not configured");
+	ERROR("Try --with-libcr option to 'configure' script");
+	return -1;
+}
+#endif
diff --git a/src/lxc/state.c b/src/lxc/state.c
index b29ae09..1e4c7e1 100644
--- a/src/lxc/state.c
+++ b/src/lxc/state.c
@@ -167,4 +167,3 @@ extern int lxc_state_callback(int fd, struct lxc_request *request,
 out:
 	return ret;
 }
-
-- 
1.6.0.4

Re: [PATCH 0/6][v3][lxc] Link LXC with USERCR

[Cedric Le Goater]

Hello Suka,

On 04/01/2010 03:55 AM, Sukadev Bhattiprolu wrote:

> Resending patches after fixing whitespace damage and the typo in
> configure.ac. Add a section to patch-0 (below) to describe the
> kernel build (commit-id, config tokens etc).

we plan to mergepatch 2,3,4 in lxc mainline real soon, maybe not as such,
but they won't be in the way anymore. For the moment, please keep 1,5,6 but
make it one patch. no need to resend them in a scattered mode.

> 1. Build C/R-enabled Linux kernel
>
> 	$ cd /root
>
> 	$ git-clone  git://www.linux-cr.org/pub/git/linux-cr.git linux-cr
>
> 	$ cd linux-cr
>
> 	$ git-checkout ckpt-v20-dev
>
> 		I tested with commit 3522c57a9ec6f08a129a78322318abcb4467db28
> 		as HEAD.

we need to work on a better delivery scheme. I don't think a commit id is not
the most efficient way to dialog when integrating components. We'd rather use
a tarball or a patch on an identified kernel.

> 	# After every reboot, ensure '-o newinstance' mount option to /dev/pts
> 	  works (see Documentation/filesystems/devpts.txt for details). In
> 	  short, run following commands on each reboot:
>
> 		$ rm /dev/ptmx
>
> 		$ ln -s pts/ptmx /dev/ptmx
>
> 		$ chmod 666 /dev/pts/ptmx

can not it be integrated in lxc as a patch ?

> 1. Build USERCR
>
> 	$ cd /root
>
> 	$ git-clone git://git.ncl.cs.columbia.edu/pub/git/user-cr.git user-cr
>
> 	$ cd user-cr
>
> 	$ git-checkout ckpt-v20-dev
>
> 	  	Tested with commit e275f77e4a82d228c1df14dbeb691342e32cdac2
> 		as HEAD.

IMO, tarball and version would be better.

> 	# Apply following two patches:
>
> 	https://lists.linux-foundation.org/pipermail/containers/2010-March/024037.html
> 	https://lists.linux-foundation.org/pipermail/containers/2010-March/024038.html

do you plan to merge them ? if not why ?

> 	$ KERNELSRC=/root/linux-cr make
>
> 		Build USERCR by pointing to corresponding kernel-source.
> 		This should create restart.o and checkpoint.o needed by LXC.
>
> 		I did not need to, but you may need to compile checkpoint.o
> 		and restart.o with -fPIC compiler option.

this needs to be worked out also to be more developer friendly.

if compilation is a pain, we will put some effort into it once but be reluctant to
do it twice. That's what happened to me this time. I didn't compile and let daniel
spent nearly one day to make your environment work ...

Thanks,

C.

Re: [PATCH 0/6][v3][lxc] Link LXC with USERCR

[Sukadev Bhattiprolu]

Cedric Le Goater [clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org] wrote:
> Hello Suka,
>
> On 04/01/2010 03:55 AM, Sukadev Bhattiprolu wrote:
>
>> Resending patches after fixing whitespace damage and the typo in
>> configure.ac. Add a section to patch-0 (below) to describe the
>> kernel build (commit-id, config tokens etc).
>
> we plan to mergepatch 2,3,4 in lxc mainline real soon, maybe not as such,
> but they won't be in the way anymore. For the moment, please keep 1,5,6 but
> make it one patch. no need to resend them in a scattered mode.

Ok. Can you apply 1,5,6 to a separate branch, say usercr-v1, so its easier
for others to pull ? When I have some major changes I can resend the patches
merged up to latest LXC and maybe we can then have usercr-v2 and so on.

>
>> 1. Build C/R-enabled Linux kernel
>>
>> 	$ cd /root
>>
>> 	$ git-clone  git://www.linux-cr.org/pub/git/linux-cr.git linux-cr
>>
>> 	$ cd linux-cr
>>
>> 	$ git-checkout ckpt-v20-dev
>>
>> 		I tested with commit 3522c57a9ec6f08a129a78322318abcb4467db28
>> 		as HEAD.
>
> we need to work on a better delivery scheme. I don't think a commit id is not
> the most efficient way to dialog when integrating components. We'd rather use
> a tarball or a patch on an identified kernel.

Hmm, I thought we could just make a branch on the commit-id and apply
any additional patches there and build.

I will base it on a closed-release next time like ckpt-v20-rc1 rather
than an open one like ckpt-v20-dev.

>
>> 	# After every reboot, ensure '-o newinstance' mount option to /dev/pts
>> 	  works (see Documentation/filesystems/devpts.txt for details). In
>> 	  short, run following commands on each reboot:
>>
>> 		$ rm /dev/ptmx
>>
>> 		$ ln -s pts/ptmx /dev/ptmx
>>
>> 		$ chmod 666 /dev/pts/ptmx
>
> can not it be integrated in lxc as a patch ?

This is an OS boot/setup issue, so its probably easier to put this in
/etc/rc.local or some such file.

>
>> 1. Build USERCR
>>
>> 	$ cd /root
>>
>> 	$ git-clone git://git.ncl.cs.columbia.edu/pub/git/user-cr.git user-cr
>>
>> 	$ cd user-cr
>>
>> 	$ git-checkout ckpt-v20-dev
>>
>> 	  	Tested with commit e275f77e4a82d228c1df14dbeb691342e32cdac2
>> 		as HEAD.
>
> IMO, tarball and version would be better.
>
>> 	# Apply following two patches:
>>
>> 	https://lists.linux-foundation.org/pipermail/containers/2010-March/024037.html
>> 	https://lists.linux-foundation.org/pipermail/containers/2010-March/024038.html
>
> do you plan to merge them ? if not why ?

Yes. They have been merged now. There maybe a few days delay between my
posting and the merging and I pointed to the link in case someone wants
to try before they are merged.
>
>> 	$ KERNELSRC=/root/linux-cr make
>>
>> 		Build USERCR by pointing to corresponding kernel-source.
>> 		This should create restart.o and checkpoint.o needed by LXC.
>>
>> 		I did not need to, but you may need to compile checkpoint.o
>> 		and restart.o with -fPIC compiler option.
>
> this needs to be worked out also to be more developer friendly.
>
> if compilation is a pain, we will put some effort into it once but be reluctant to
> do it twice. That's what happened to me this time. I didn't compile and let daniel
> spent nearly one day to make your environment work ...

Agree and am planning to write up these instructions in a wiki and
hopefully we can simplify the process over time. I tested several times
on my environment before posting :-) 

BTW, Is fPIC needed always ?

Sukadev

Re: [PATCH 0/6][v3][lxc] Link LXC with USERCR

[Oren Laadan]

Sukadev Bhattiprolu wrote:
> Cedric Le Goater [clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org] wrote:
>> Hello Suka,
>>
>> On 04/01/2010 03:55 AM, Sukadev Bhattiprolu wrote:
>>
>>> Resending patches after fixing whitespace damage and the typo in
>>> configure.ac. Add a section to patch-0 (below) to describe the
>>> kernel build (commit-id, config tokens etc).
>> we plan to mergepatch 2,3,4 in lxc mainline real soon, maybe not as such,
>> but they won't be in the way anymore. For the moment, please keep 1,5,6 but
>> make it one patch. no need to resend them in a scattered mode.
> 
> Ok. Can you apply 1,5,6 to a separate branch, say usercr-v1, so its easier
> for others to pull ? When I have some major changes I can resend the patches
> merged up to latest LXC and maybe we can then have usercr-v2 and so on.
> 
>>> 1. Build C/R-enabled Linux kernel
>>>
>>> 	$ cd /root
>>>
>>> 	$ git-clone  git://www.linux-cr.org/pub/git/linux-cr.git linux-cr
>>>
>>> 	$ cd linux-cr
>>>
>>> 	$ git-checkout ckpt-v20-dev
>>>
>>> 		I tested with commit 3522c57a9ec6f08a129a78322318abcb4467db28
>>> 		as HEAD.
>> we need to work on a better delivery scheme. I don't think a commit id is not
>> the most efficient way to dialog when integrating components. We'd rather use
>> a tarball or a patch on an identified kernel.
> 
> Hmm, I thought we could just make a branch on the commit-id and apply
> any additional patches there and build.
> 
> I will base it on a closed-release next time like ckpt-v20-rc1 rather
> than an open one like ckpt-v20-dev.

The stable and -rc versions are tagged with cr-vXX[-rcY].

For example, current version is cr-v21-rc2 (both linux-cr and user-cr)

Oren.