[PATCH v5 0/9] nfsdcld: add a daemon to track NFSv4 client names on stable storage

[PATCH v5 0/9] nfsdcld: add a daemon to track NFSv4 client names on stable storage

[Jeff Layton]

This patchset is the userspace portion of the knfsd client name tracking
overhaul. See this patch series for an explanation:

    nfsd: overhaul the client name tracking code

This version scales back some of the changes in the last set. Since
then, it's become clear that I'm not entirely certain what changes we'll
need in order to handle this in a clustered configuration. Instead of
trying to anticipate those changes, I've decided just to focus on the
single-server case and leave any other changes for later once it becomes
more clear what (if anything) is needed.

The main changes from the last set are:

- the "remove" upcall has been readded

- the "init" upcall has been removed

- the "check" and "update" upcalls no longer return a client index
  to the kernel

- the pipe open and reopen code has been overhauled and consolidated.
  Instead of looping indefinitely when trying to reopen the file, it will
  abort the daemon if the open fails with anything besides ENOENT. If
  the pipe isn't present when the daemon starts, it now uses the inotify
  event callback to handle the opening.

Jeff Layton (9):
  nfsdcld: add client tracking daemon stub
  nfsdcld: add autoconf goop for sqlite
  nfsdcld: add routines for a sqlite backend database
  nfsdcld: add remove functionality
  nfsdcld: add check/update functionality
  nfsdcld: add function to remove unreclaimed client records
  nfsdcld: make it watch for inotify events in the containing directory
  nfsdcld: add a manpage for nfsdcld
  nfsdcld: update the README

 README                    |   25 ++-
 aclocal/libsqlite3.m4     |   33 +++
 configure.ac              |   21 ++
 utils/Makefile.am         |    4 +
 utils/nfsdcld/Makefile.am |   14 ++
 utils/nfsdcld/nfsdcld.c   |  528 +++++++++++++++++++++++++++++++++++++++++++++
 utils/nfsdcld/nfsdcld.man |  180 +++++++++++++++
 utils/nfsdcld/nfsdcld.pod |   67 ++++++
 utils/nfsdcld/sqlite.c    |  390 +++++++++++++++++++++++++++++++++
 utils/nfsdcld/sqlite.h    |   29 +++
 10 files changed, 1289 insertions(+), 2 deletions(-)
 create mode 100644 aclocal/libsqlite3.m4
 create mode 100644 utils/nfsdcld/Makefile.am
 create mode 100644 utils/nfsdcld/nfsdcld.c
 create mode 100644 utils/nfsdcld/nfsdcld.man
 create mode 100644 utils/nfsdcld/nfsdcld.pod
 create mode 100644 utils/nfsdcld/sqlite.c
 create mode 100644 utils/nfsdcld/sqlite.h

-- 
1.7.7.6

[PATCH v5 1/9] nfsdcld: add client tracking daemon stub

[Jeff Layton]

This program opens and "listens" on the new nfsd/cld rpc_pipefs pipe.
The code here doesn't actually do anything on stable storage yet. That
will be added in a later patch.

The patch also adds a autoconf enable switch for the new daemon that
defaults to "no", and a test for the upcall description header file.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 configure.ac              |   14 +++
 utils/Makefile.am         |    4 +
 utils/nfsdcld/Makefile.am |   14 +++
 utils/nfsdcld/nfsdcld.c   |  275 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 307 insertions(+), 0 deletions(-)
 create mode 100644 utils/nfsdcld/Makefile.am
 create mode 100644 utils/nfsdcld/nfsdcld.c

diff --git a/configure.ac b/configure.ac
index 920e8da..d50e54e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -185,6 +185,12 @@ else
 	AM_CONDITIONAL(MOUNT_CONFIG, [test "$enable_mount" = "yes"])
 fi
 
+AC_ARG_ENABLE(nfsdcld,
+	[AC_HELP_STRING([--enable-nfsdcld],
+			[Create nfsdcld NFSv4 clientid tracking daemon. <:@default=no@:>@])],
+	enable_nfsdcld=$enableval,
+	enable_nfsdcld="no")
+
 dnl Check for TI-RPC library and headers
 AC_LIBTIRPC
 
@@ -260,6 +266,13 @@ if test "$enable_nfsv4" = yes; then
   dnl check for the keyutils libraries and headers
   AC_KEYUTILS
 
+  if test "$enable_nfsdcld" = "yes"; then
+    AC_CHECK_HEADERS([linux/nfsd/cld.h], ,
+		     AC_MSG_ERROR([Cannot find header needed for nfsdcld]))
+  fi
+
+  AM_CONDITIONAL(CONFIG_NFSDCLD, [test "$enable_nfsdcld" = "yes" ])
+
   dnl librpcsecgss already has a dependency on libgssapi,
   dnl but we need to make sure we get the right version
   if test "$enable_gss" = yes; then
@@ -460,6 +473,7 @@ AC_CONFIG_FILES([
 	tools/nfs-iostat/Makefile
 	utils/Makefile
 	utils/blkmapd/Makefile
+	utils/nfsdcld/Makefile
 	utils/exportfs/Makefile
 	utils/gssd/Makefile
 	utils/idmapd/Makefile
diff --git a/utils/Makefile.am b/utils/Makefile.am
index d074b85..5df7ca7 100644
--- a/utils/Makefile.am
+++ b/utils/Makefile.am
@@ -21,6 +21,10 @@ if CONFIG_MOUNT
 OPTDIRS += mount
 endif
 
+if CONFIG_NFSDCLD
+OPTDIRS += nfsdcld
+endif
+
 SUBDIRS = \
 	exportfs \
 	mountd \
diff --git a/utils/nfsdcld/Makefile.am b/utils/nfsdcld/Makefile.am
new file mode 100644
index 0000000..ed7ed42
--- /dev/null
+++ b/utils/nfsdcld/Makefile.am
@@ -0,0 +1,14 @@
+## Process this file with automake to produce Makefile.in
+
+#man8_MANS	= nfsdcld.man
+#EXTRA_DIST = $(man8_MANS)
+
+AM_CFLAGS	+= -D_LARGEFILE64_SOURCE
+sbin_PROGRAMS	= nfsdcld
+
+nfsdcld_SOURCES = nfsdcld.c
+
+nfsdcld_LDADD = ../../support/nfs/libnfs.a $(LIBEVENT)
+
+MAINTAINERCLEANFILES = Makefile.in
+
diff --git a/utils/nfsdcld/nfsdcld.c b/utils/nfsdcld/nfsdcld.c
new file mode 100644
index 0000000..7a83fbe
--- /dev/null
+++ b/utils/nfsdcld/nfsdcld.c
@@ -0,0 +1,275 @@
+/*
+ * nfsdcld.c -- NFSv4 client name tracking daemon
+ *
+ * Copyright (C) 2011  Red Hat, Jeff Layton <jlayton@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor,
+ * Boston, MA 02110-1301, USA.
+ */
+
+#ifdef HAVE_CONFIG_H
+#include "config.h"
+#endif /* HAVE_CONFIG_H */
+
+#include <errno.h>
+#include <event.h>
+#include <stdbool.h>
+#include <getopt.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <linux/nfsd/cld.h>
+
+#include "xlog.h"
+#include "nfslib.h"
+
+#ifndef PIPEFS_DIR
+#define PIPEFS_DIR NFS_STATEDIR "/rpc_pipefs"
+#endif
+
+#define DEFAULT_CLD_PATH	PIPEFS_DIR "/nfsd/cld"
+
+#define UPCALL_VERSION		1
+
+/* private data structures */
+struct cld_client {
+	int			cl_fd;
+	struct event		cl_event;
+	struct cld_msg	cl_msg;
+};
+
+/* global variables */
+static char *pipepath = DEFAULT_CLD_PATH;
+
+static struct option longopts[] =
+{
+	{ "help", 0, NULL, 'h' },
+	{ "foreground", 0, NULL, 'F' },
+	{ "debug", 0, NULL, 'd' },
+	{ "pipe", 1, NULL, 'p' },
+	{ NULL, 0, 0, 0 },
+};
+
+/* forward declarations */
+static void cldcb(int UNUSED(fd), short which, void *data);
+
+static void
+usage(char *progname)
+{
+	printf("Usage:\n");
+	printf("%s [ -hFd ] [ -p pipe ]\n", progname);
+}
+
+static int
+cld_pipe_open(struct cld_client *clnt)
+{
+	int fd;
+
+	xlog(D_GENERAL, "%s: opening upcall pipe %s", __func__, pipepath);
+	fd = open(pipepath, O_RDWR, 0);
+	if (fd < 0) {
+		xlog(L_ERROR, "%s: open of %s failed: %m", __func__, pipepath);
+		return -errno;
+	}
+
+	if (clnt->cl_event.ev_flags & EVLIST_INIT)
+		event_del(&clnt->cl_event);
+	if (clnt->cl_fd >= 0)
+		close(clnt->cl_fd);
+
+	clnt->cl_fd = fd;
+	event_set(&clnt->cl_event, clnt->cl_fd, EV_READ, cldcb, clnt);
+	/* event_add is done by the caller */
+	return 0;
+}
+
+static int
+cld_pipe_init(struct cld_client *clnt)
+{
+	int ret;
+
+	clnt->cl_fd = -1;
+	ret = cld_pipe_open(clnt);
+	if (ret)
+		return ret;
+
+	event_add(&clnt->cl_event, NULL);
+	return 0;
+}
+
+static void
+cld_not_implemented(struct cld_client *clnt)
+{
+	int ret;
+	ssize_t bsize, wsize;
+	struct cld_msg *cmsg = &clnt->cl_msg;
+
+	xlog(D_GENERAL, "%s: downcalling with not implemented error", __func__);
+
+	/* set up reply */
+	cmsg->cm_status = -EOPNOTSUPP;
+
+	bsize = sizeof(*cmsg);
+
+	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
+	if (wsize != bsize)
+		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
+			 __func__, wsize);
+
+	/* reopen pipe, just to be sure */
+	ret = cld_pipe_open(clnt);
+	if (ret) {
+		xlog(L_FATAL, "%s: unable to reopen pipe: %d", __func__, ret);
+		exit(ret);
+	}
+}
+
+static void
+cld_create(struct cld_client *clnt)
+{
+	int ret;
+	ssize_t bsize, wsize;
+	struct cld_msg *cmsg = &clnt->cl_msg;
+
+	xlog(D_GENERAL, "%s: create client record", __func__);
+
+	/* FIXME: create client record on storage here */
+
+	/* set up reply */
+	cmsg->cm_status = 0;
+
+	bsize = sizeof(*cmsg);
+
+	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
+	if (wsize != bsize) {
+		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
+			 __func__, wsize);
+		ret = cld_pipe_open(clnt);
+		if (ret) {
+			xlog(L_FATAL, "%s: unable to reopen pipe: %d",
+					__func__, ret);
+			exit(ret);
+		}
+	}
+}
+
+static void
+cldcb(int UNUSED(fd), short which, void *data)
+{
+	ssize_t len;
+	struct cld_client *clnt = data;
+	struct cld_msg *cmsg = &clnt->cl_msg;
+
+	if (which != EV_READ)
+		goto out;
+
+	len = atomicio(read, clnt->cl_fd, cmsg, sizeof(*cmsg));
+	if (len <= 0) {
+		xlog(L_ERROR, "%s: pipe read failed: %m", __func__);
+		cld_pipe_open(clnt);
+		goto out;
+	}
+
+	if (cmsg->cm_vers != UPCALL_VERSION) {
+		xlog(L_ERROR, "%s: unsupported upcall version: %hu",
+				cmsg->cm_vers);
+		cld_pipe_open(clnt);
+		goto out;
+	}
+
+	switch(cmsg->cm_cmd) {
+	case Cld_Create:
+		cld_create(clnt);
+		break;
+	default:
+		xlog(L_WARNING, "%s: command %u is not yet implemented",
+				__func__, cmsg->cm_cmd);
+		cld_not_implemented(clnt);
+	}
+out:
+	event_add(&clnt->cl_event, NULL);
+}
+
+int
+main(int argc, char **argv)
+{
+	char arg;
+	int rc = 0;
+	bool foreground = false;
+	char *progname;
+	struct cld_client clnt;
+
+	memset(&clnt, 0, sizeof(clnt));
+
+	progname = strdup(basename(argv[0]));
+	if (!progname) {
+		fprintf(stderr, "%s: unable to allocate memory.\n", argv[0]);
+		return 1;
+	}
+
+	event_init();
+	xlog_syslog(0);
+	xlog_stderr(1);
+
+	/* process command-line options */
+	while ((arg = getopt_long(argc, argv, "hdFp:", longopts,
+				  NULL)) != EOF) {
+		switch (arg) {
+		case 'd':
+			xlog_config(D_ALL, 1);
+			break;
+		case 'F':
+			foreground = true;
+			break;
+		case 'p':
+			pipepath = optarg;
+			break;
+		default:
+			usage(progname);
+			return 0;
+		}
+	}
+
+
+	xlog_open(progname);
+	if (!foreground) {
+		xlog_syslog(1);
+		xlog_stderr(0);
+		rc = daemon(0, 0);
+		if (rc) {
+			xlog(L_ERROR, "Unable to daemonize: %m");
+			goto out;
+		}
+	}
+
+	/* set up storage db */
+
+	/* set up event handler */
+	rc = cld_pipe_init(&clnt);
+	if (rc)
+		goto out;
+
+	xlog(D_GENERAL, "%s: Starting event dispatch handler.", __func__);
+	rc = event_dispatch();
+	if (rc < 0)
+		xlog(L_ERROR, "%s: event_dispatch failed: %m", __func__);
+
+	close(clnt.cl_fd);
+out:
+	free(progname);
+	return rc;
+}
-- 
1.7.7.6

[PATCH v5 2/9] nfsdcld: add autoconf goop for sqlite

[Jeff Layton]

Mostly cribbed from Chuck Lever's new-statd rewrite a few years ago...

This adds an autoconf test for the sqlite3 library and includes. If
they're not working properly and nfsdcld was enabled, then configure
will error out.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 aclocal/libsqlite3.m4 |   33 +++++++++++++++++++++++++++++++++
 configure.ac          |    7 +++++++
 2 files changed, 40 insertions(+), 0 deletions(-)
 create mode 100644 aclocal/libsqlite3.m4

diff --git a/aclocal/libsqlite3.m4 b/aclocal/libsqlite3.m4
new file mode 100644
index 0000000..73d1e46
--- /dev/null
+++ b/aclocal/libsqlite3.m4
@@ -0,0 +1,33 @@
+dnl Checks for matching sqlite3 header and library, and
+dnl sufficient sqlite3 version.
+dnl
+AC_DEFUN([AC_SQLITE3_VERS], [
+  AC_CHECK_HEADERS([sqlite3.h], ,)
+
+  dnl look for the library; do not add to LIBS if found
+  AC_CHECK_LIB([sqlite3], [sqlite3_libversion_number], [LIBSQLITE=-lsqlite3], ,)
+  AC_SUBST(LIBSQLITE)
+
+  AC_MSG_CHECKING(for suitable sqlite3 version)
+
+  AC_CACHE_VAL([libsqlite3_cv_is_recent],
+   [
+    saved_LIBS="$LIBS"
+    LIBS=-lsqlite3
+    AC_TRY_RUN([
+	#include <stdio.h>
+	#include <sqlite3.h>
+	int main()
+	{
+		int vers = sqlite3_libversion_number();
+
+		return vers != SQLITE_VERSION_NUMBER ||
+			vers < 3003000;
+	}
+       ], [libsqlite3_cv_is_recent=yes], [libsqlite3_cv_is_recent=no],
+       [libsqlite3_cv_is_recent=unknown])
+    LIBS="$saved_LIBS"])
+
+  AC_MSG_RESULT($libsqlite3_cv_is_recent)
+  AM_CONDITIONAL(CONFIG_SQLITE3, [test "$libsqlite3_cv_is_recent" = "yes"])
+])dnl
diff --git a/configure.ac b/configure.ac
index d50e54e..134b609 100644
--- a/configure.ac
+++ b/configure.ac
@@ -266,9 +266,16 @@ if test "$enable_nfsv4" = yes; then
   dnl check for the keyutils libraries and headers
   AC_KEYUTILS
 
+  dnl Check for sqlite3
+  AC_SQLITE3_VERS
+
   if test "$enable_nfsdcld" = "yes"; then
     AC_CHECK_HEADERS([linux/nfsd/cld.h], ,
 		     AC_MSG_ERROR([Cannot find header needed for nfsdcld]))
+
+    if test "$libsqlite3_cv_is_recent" != "yes" ; then
+      AC_MSG_ERROR([nfsdcld requires sqlite3])
+    fi
   fi
 
   AM_CONDITIONAL(CONFIG_NFSDCLD, [test "$enable_nfsdcld" = "yes" ])
-- 
1.7.7.6

[PATCH v5 3/9] nfsdcld: add routines for a sqlite backend database

[Jeff Layton]

Rather than roll our own "storage engine", use sqlite instead. It fits
the bill nicely as it does:

- durable on-disk storage
- the ability to constrain record uniqueness
- a facility for collating and searching the host records

...it does add a build dependency to nfs-utils, but almost all modern
distros provide those packages.

The current incarnation of this code dynamically links against a
provided sqlite library, but we could also consider including their
single-file "amalgamation" to reduce dependencies (though with all
the caveats that that entails).

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 utils/nfsdcld/Makefile.am |    4 +-
 utils/nfsdcld/nfsdcld.c   |   24 +++-
 utils/nfsdcld/sqlite.c    |  253 +++++++++++++++++++++++++++++++++++++++++++++
 utils/nfsdcld/sqlite.h    |   26 +++++
 4 files changed, 299 insertions(+), 8 deletions(-)
 create mode 100644 utils/nfsdcld/sqlite.c
 create mode 100644 utils/nfsdcld/sqlite.h

diff --git a/utils/nfsdcld/Makefile.am b/utils/nfsdcld/Makefile.am
index ed7ed42..8e4f2ab 100644
--- a/utils/nfsdcld/Makefile.am
+++ b/utils/nfsdcld/Makefile.am
@@ -6,9 +6,9 @@
 AM_CFLAGS	+= -D_LARGEFILE64_SOURCE
 sbin_PROGRAMS	= nfsdcld
 
-nfsdcld_SOURCES = nfsdcld.c
+nfsdcld_SOURCES = nfsdcld.c sqlite.c
 
-nfsdcld_LDADD = ../../support/nfs/libnfs.a $(LIBEVENT)
+nfsdcld_LDADD = ../../support/nfs/libnfs.a $(LIBEVENT) $(LIBSQLITE)
 
 MAINTAINERCLEANFILES = Makefile.in
 
diff --git a/utils/nfsdcld/nfsdcld.c b/utils/nfsdcld/nfsdcld.c
index 7a83fbe..15746e6 100644
--- a/utils/nfsdcld/nfsdcld.c
+++ b/utils/nfsdcld/nfsdcld.c
@@ -36,6 +36,7 @@
 
 #include "xlog.h"
 #include "nfslib.h"
+#include "sqlite.h"
 
 #ifndef PIPEFS_DIR
 #define PIPEFS_DIR NFS_STATEDIR "/rpc_pipefs"
@@ -61,6 +62,7 @@ static struct option longopts[] =
 	{ "foreground", 0, NULL, 'F' },
 	{ "debug", 0, NULL, 'd' },
 	{ "pipe", 1, NULL, 'p' },
+	{ "storagedir", 1, NULL, 's' },
 	{ NULL, 0, 0, 0 },
 };
 
@@ -71,7 +73,7 @@ static void
 usage(char *progname)
 {
 	printf("Usage:\n");
-	printf("%s [ -hFd ] [ -p pipe ]\n", progname);
+	printf("%s [ -hFd ] [ -p pipe ] [ -s dir ]\n", progname);
 }
 
 static int
@@ -145,15 +147,16 @@ cld_create(struct cld_client *clnt)
 	ssize_t bsize, wsize;
 	struct cld_msg *cmsg = &clnt->cl_msg;
 
-	xlog(D_GENERAL, "%s: create client record", __func__);
+	xlog(D_GENERAL, "%s: create client record.", __func__);
 
-	/* FIXME: create client record on storage here */
+	ret = sqlite_insert_client(cmsg->cm_u.cm_name.cn_id,
+				   cmsg->cm_u.cm_name.cn_len);
 
-	/* set up reply */
-	cmsg->cm_status = 0;
+	cmsg->cm_status = ret ? -EREMOTEIO : ret;
 
 	bsize = sizeof(*cmsg);
 
+	xlog(D_GENERAL, "Doing downcall with status %d", cmsg->cm_status);
 	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
 	if (wsize != bsize) {
 		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
@@ -211,6 +214,7 @@ main(int argc, char **argv)
 	int rc = 0;
 	bool foreground = false;
 	char *progname;
+	char *storagedir = NULL;
 	struct cld_client clnt;
 
 	memset(&clnt, 0, sizeof(clnt));
@@ -226,7 +230,7 @@ main(int argc, char **argv)
 	xlog_stderr(1);
 
 	/* process command-line options */
-	while ((arg = getopt_long(argc, argv, "hdFp:", longopts,
+	while ((arg = getopt_long(argc, argv, "hdFp:s:", longopts,
 				  NULL)) != EOF) {
 		switch (arg) {
 		case 'd':
@@ -238,6 +242,9 @@ main(int argc, char **argv)
 		case 'p':
 			pipepath = optarg;
 			break;
+		case 's':
+			storagedir = optarg;
+			break;
 		default:
 			usage(progname);
 			return 0;
@@ -257,6 +264,11 @@ main(int argc, char **argv)
 	}
 
 	/* set up storage db */
+	rc = sqlite_maindb_init(storagedir);
+	if (rc) {
+		xlog(L_ERROR, "Failed to open main database: %d", rc);
+		goto out;
+	}
 
 	/* set up event handler */
 	rc = cld_pipe_init(&clnt);
diff --git a/utils/nfsdcld/sqlite.c b/utils/nfsdcld/sqlite.c
new file mode 100644
index 0000000..f70568e
--- /dev/null
+++ b/utils/nfsdcld/sqlite.c
@@ -0,0 +1,253 @@
+/*
+ * Copyright (C) 2011  Red Hat, Jeff Layton <jlayton@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor,
+ * Boston, MA 02110-1301, USA.
+ */
+
+/*
+ * Explanation:
+ *
+ * This file contains the code to manage the sqlite backend database for the
+ * clstated upcall daemon.
+ *
+ * The main database is called main.sqlite and contains the following tables:
+ *
+ * parameters: simple key/value pairs for storing database info
+ *
+ * clients: one column containing a BLOB with the as sent by the client
+ * 	    and a timestamp (in epoch seconds) of when the record was
+ * 	    established
+ *
+ * FIXME: should we also record the fsid being accessed?
+ */
+
+#ifdef HAVE_CONFIG_H
+#include "config.h"
+#endif /* HAVE_CONFIG_H */
+
+#include <errno.h>
+#include <event.h>
+#include <stdbool.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <sqlite3.h>
+#include <linux/limits.h>
+
+#include "xlog.h"
+
+#define CLD_SQLITE_SCHEMA_VERSION 1
+
+#ifndef CLD_SQLITE_TOPDIR
+#define CLD_SQLITE_TOPDIR NFS_STATEDIR "/nfsdcld"
+#endif
+
+/* in milliseconds */
+#define CLD_SQLITE_BUSY_TIMEOUT 10000
+
+/* private data structures */
+
+/* global variables */
+
+/* top level DB directory */
+static char *sqlite_topdir;
+
+/* reusable pathname and sql command buffer */
+static char buf[PATH_MAX];
+
+/* global database handle */
+static sqlite3 *dbh;
+
+/* forward declarations */
+
+/* make a directory, ignoring EEXIST errors unless it's not a directory */
+static int
+mkdir_if_not_exist(char *dirname)
+{
+	int ret;
+	struct stat statbuf;
+
+	ret = mkdir(dirname, S_IRWXU);
+	if (ret && errno != EEXIST)
+		return -errno;
+
+	ret = stat(dirname, &statbuf);
+	if (ret)
+		return -errno;
+
+	if (!S_ISDIR(statbuf.st_mode))
+		ret = -ENOTDIR;
+
+	return ret;
+}
+
+/*
+ * Open the "main" database, and attempt to initialize it by creating the
+ * parameters table and inserting the schema version into it. Ignore any errors
+ * from that, and then attempt to select the version out of it again. If the
+ * version appears wrong, then assume that the DB is corrupt or has been
+ * upgraded, and return an error. If all of that works, then attempt to create
+ * the "clients" table.
+ */
+int
+sqlite_maindb_init(char *topdir)
+{
+	int ret;
+	char *err = NULL;
+	sqlite3_stmt *stmt = NULL;
+
+	sqlite_topdir = topdir ? topdir : CLD_SQLITE_TOPDIR;
+
+	ret = mkdir_if_not_exist(sqlite_topdir);
+	if (ret)
+		return ret;
+
+	ret = snprintf(buf, PATH_MAX - 1, "%s/main.sqlite", sqlite_topdir);
+	if (ret < 0)
+		return ret;
+
+	buf[PATH_MAX - 1] = '\0';
+
+	ret = sqlite3_open(buf, &dbh);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "Unable to open main database: %d", ret);
+		return ret;
+	}
+
+	ret = sqlite3_busy_timeout(dbh, CLD_SQLITE_BUSY_TIMEOUT);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "Unable to set sqlite busy timeout: %d", ret);
+		goto out_err;
+	}
+
+	/* Try to create table */
+	ret = sqlite3_exec(dbh, "CREATE TABLE IF NOT EXISTS parameters "
+				"(key TEXT PRIMARY KEY, value TEXT);",
+				NULL, NULL, &err);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "Unable to create parameter table: %d", ret);
+		goto out_err;
+	}
+
+	/* insert version into table -- ignore error if it fails */
+	ret = snprintf(buf, sizeof(buf),
+		       "INSERT OR IGNORE INTO parameters values (\"version\", "
+		       "\"%d\");", CLD_SQLITE_SCHEMA_VERSION);
+	if (ret < 0) {
+		goto out_err;
+	} else if ((size_t)ret >= sizeof(buf)) {
+		ret = -EINVAL;
+		goto out_err;
+	}
+
+	ret = sqlite3_exec(dbh, (const char *)buf, NULL, NULL, &err);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "Unable to insert into parameter table: %d",
+				ret);
+		goto out_err;
+	}
+
+	ret = sqlite3_prepare_v2(dbh,
+		"SELECT value FROM parameters WHERE key == \"version\";",
+		 -1, &stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "Unable to prepare select statement: %d", ret);
+		goto out_err;
+	}
+
+	/* check schema version */
+	ret = sqlite3_step(stmt);
+	if (ret != SQLITE_ROW) {
+		xlog(L_ERROR, "Select statement execution failed: %s",
+				sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	/* process SELECT result */
+	ret = sqlite3_column_int(stmt, 0);
+	if (ret != CLD_SQLITE_SCHEMA_VERSION) {
+		xlog(L_ERROR, "Unsupported database schema version! "
+			"Expected %d, got %d.",
+			CLD_SQLITE_SCHEMA_VERSION, ret);
+		ret = -EINVAL;
+		goto out_err;
+	}
+
+	/* now create the "clients" table */
+	ret = sqlite3_exec(dbh, "CREATE TABLE IF NOT EXISTS clients "
+				"(id BLOB PRIMARY KEY, time INTEGER);",
+				NULL, NULL, &err);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "Unable to create clients table: %s", err);
+		goto out_err;
+	}
+
+	sqlite3_free(err);
+	sqlite3_finalize(stmt);
+	return 0;
+
+out_err:
+	if (err) {
+		xlog(L_ERROR, "sqlite error: %s", err);
+		sqlite3_free(err);
+	}
+	sqlite3_finalize(stmt);
+	sqlite3_close(dbh);
+	return ret;
+}
+
+/*
+ * Create a client record
+ *
+ * Returns a non-zero sqlite error code, or SQLITE_OK (aka 0)
+ */
+int
+sqlite_insert_client(const unsigned char *clname, const size_t namelen)
+{
+	int ret;
+	sqlite3_stmt *stmt = NULL;
+
+	ret = sqlite3_prepare_v2(dbh, "INSERT OR REPLACE INTO clients VALUES "
+				      "(?, strftime('%s', 'now'));", -1,
+					&stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: insert statement prepare failed: %s",
+			__func__, sqlite3_errmsg(dbh));
+		return ret;
+	}
+
+	ret = sqlite3_bind_blob(stmt, 1, (const void *)clname, namelen,
+				SQLITE_STATIC);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: bind blob failed: %s", __func__,
+				sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	ret = sqlite3_step(stmt);
+	if (ret == SQLITE_DONE)
+		ret = SQLITE_OK;
+	else
+		xlog(L_ERROR, "%s: unexpected return code from insert: %s",
+				__func__, sqlite3_errmsg(dbh));
+
+out_err:
+	xlog(D_GENERAL, "%s: returning %d", __func__, ret);
+	sqlite3_finalize(stmt);
+	return ret;
+}
diff --git a/utils/nfsdcld/sqlite.h b/utils/nfsdcld/sqlite.h
new file mode 100644
index 0000000..ba4c213
--- /dev/null
+++ b/utils/nfsdcld/sqlite.h
@@ -0,0 +1,26 @@
+/*
+ * Copyright (C) 2011  Red Hat, Jeff Layton <jlayton@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor,
+ * Boston, MA 02110-1301, USA.
+ */
+
+#ifndef _SQLITE_H_
+#define _SQLITE_H_
+
+int sqlite_maindb_init(char *topdir);
+int sqlite_insert_client(const unsigned char *clname, const size_t namelen);
+
+#endif /* _SQLITE_H */
-- 
1.7.7.6

[PATCH v5 4/9] nfsdcld: add remove functionality

[Jeff Layton]

Allow the kernel to ask for removal of a client record.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 utils/nfsdcld/nfsdcld.c |   34 ++++++++++++++++++++++++++++++++++
 utils/nfsdcld/sqlite.c  |   36 ++++++++++++++++++++++++++++++++++++
 utils/nfsdcld/sqlite.h  |    1 +
 3 files changed, 71 insertions(+), 0 deletions(-)

diff --git a/utils/nfsdcld/nfsdcld.c b/utils/nfsdcld/nfsdcld.c
index 15746e6..ce8355a 100644
--- a/utils/nfsdcld/nfsdcld.c
+++ b/utils/nfsdcld/nfsdcld.c
@@ -171,6 +171,37 @@ cld_create(struct cld_client *clnt)
 }
 
 static void
+cld_remove(struct cld_client *clnt)
+{
+	int ret;
+	ssize_t bsize, wsize;
+	struct cld_msg *cmsg = &clnt->cl_msg;
+
+	xlog(D_GENERAL, "%s: remove client record.", __func__);
+
+	ret = sqlite_remove_client(cmsg->cm_u.cm_name.cn_id,
+				   cmsg->cm_u.cm_name.cn_len);
+
+	cmsg->cm_status = ret ? -EREMOTEIO : ret;
+
+	bsize = sizeof(*cmsg);
+
+	xlog(D_GENERAL, "%s: downcall with status %d", __func__,
+			cmsg->cm_status);
+	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
+	if (wsize != bsize) {
+		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
+			 __func__, wsize);
+		ret = cld_pipe_open(clnt);
+		if (ret) {
+			xlog(L_FATAL, "%s: unable to reopen pipe: %d",
+					__func__, ret);
+			exit(ret);
+		}
+	}
+}
+
+static void
 cldcb(int UNUSED(fd), short which, void *data)
 {
 	ssize_t len;
@@ -198,6 +229,9 @@ cldcb(int UNUSED(fd), short which, void *data)
 	case Cld_Create:
 		cld_create(clnt);
 		break;
+	case Cld_Remove:
+		cld_remove(clnt);
+		break;
 	default:
 		xlog(L_WARNING, "%s: command %u is not yet implemented",
 				__func__, cmsg->cm_cmd);
diff --git a/utils/nfsdcld/sqlite.c b/utils/nfsdcld/sqlite.c
index f70568e..a198c34 100644
--- a/utils/nfsdcld/sqlite.c
+++ b/utils/nfsdcld/sqlite.c
@@ -251,3 +251,39 @@ out_err:
 	sqlite3_finalize(stmt);
 	return ret;
 }
+
+/* Remove a client record */
+int
+sqlite_remove_client(const unsigned char *clname, const size_t namelen)
+{
+	int ret;
+	sqlite3_stmt *stmt = NULL;
+
+	ret = sqlite3_prepare_v2(dbh, "DELETE FROM clients WHERE id==?", -1,
+				 &stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: statement prepare failed: %s",
+				__func__, sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	ret = sqlite3_bind_blob(stmt, 1, (const void *)clname, namelen,
+				SQLITE_STATIC);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: bind blob failed: %s", __func__,
+				sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	ret = sqlite3_step(stmt);
+	if (ret == SQLITE_DONE)
+		ret = SQLITE_OK;
+	else
+		xlog(L_ERROR, "%s: unexpected return code from delete: %d",
+				__func__, ret);
+
+out_err:
+	xlog(D_GENERAL, "%s: returning %d", __func__, ret);
+	sqlite3_finalize(stmt);
+	return ret;
+}
diff --git a/utils/nfsdcld/sqlite.h b/utils/nfsdcld/sqlite.h
index ba4c213..425f5ba 100644
--- a/utils/nfsdcld/sqlite.h
+++ b/utils/nfsdcld/sqlite.h
@@ -22,5 +22,6 @@
 
 int sqlite_maindb_init(char *topdir);
 int sqlite_insert_client(const unsigned char *clname, const size_t namelen);
+int sqlite_remove_client(const unsigned char *clname, const size_t namelen);
 
 #endif /* _SQLITE_H */
-- 
1.7.7.6

[PATCH v5 5/9] nfsdcld: add check/update functionality

[Jeff Layton]

Add functions to check whether a client is allowed to reclaim, and
update its timestamp in the DB if so. If either the query or update
fails, then the host is not allowed to reclaim state.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 utils/nfsdcld/nfsdcld.c |   35 ++++++++++++++++++++++
 utils/nfsdcld/sqlite.c  |   73 +++++++++++++++++++++++++++++++++++++++++++++++
 utils/nfsdcld/sqlite.h  |    1 +
 3 files changed, 109 insertions(+), 0 deletions(-)

diff --git a/utils/nfsdcld/nfsdcld.c b/utils/nfsdcld/nfsdcld.c
index ce8355a..eb01d23 100644
--- a/utils/nfsdcld/nfsdcld.c
+++ b/utils/nfsdcld/nfsdcld.c
@@ -202,6 +202,38 @@ cld_remove(struct cld_client *clnt)
 }
 
 static void
+cld_check(struct cld_client *clnt)
+{
+	int ret;
+	ssize_t bsize, wsize;
+	struct cld_msg *cmsg = &clnt->cl_msg;
+
+	xlog(D_GENERAL, "%s: check client record", __func__);
+
+	ret = sqlite_check_client(cmsg->cm_u.cm_name.cn_id,
+				  cmsg->cm_u.cm_name.cn_len);
+
+	/* set up reply */
+	cmsg->cm_status = ret ? -EACCES : ret;
+
+	bsize = sizeof(*cmsg);
+
+	xlog(D_GENERAL, "%s: downcall with status %d", __func__,
+			cmsg->cm_status);
+	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
+	if (wsize != bsize) {
+		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
+			 __func__, wsize);
+		ret = cld_pipe_open(clnt);
+		if (ret) {
+			xlog(L_FATAL, "%s: unable to reopen pipe: %d",
+					__func__, ret);
+			exit(ret);
+		}
+	}
+}
+
+static void
 cldcb(int UNUSED(fd), short which, void *data)
 {
 	ssize_t len;
@@ -232,6 +264,9 @@ cldcb(int UNUSED(fd), short which, void *data)
 	case Cld_Remove:
 		cld_remove(clnt);
 		break;
+	case Cld_Check:
+		cld_check(clnt);
+		break;
 	default:
 		xlog(L_WARNING, "%s: command %u is not yet implemented",
 				__func__, cmsg->cm_cmd);
diff --git a/utils/nfsdcld/sqlite.c b/utils/nfsdcld/sqlite.c
index a198c34..01bba1a 100644
--- a/utils/nfsdcld/sqlite.c
+++ b/utils/nfsdcld/sqlite.c
@@ -287,3 +287,76 @@ out_err:
 	sqlite3_finalize(stmt);
 	return ret;
 }
+
+/*
+ * Is the given clname in the clients table? If so, then update its timestamp
+ * and return success. If the record isn't present, or the update fails, then
+ * return an error.
+ */
+int
+sqlite_check_client(const unsigned char *clname, const size_t namelen)
+{
+	int ret;
+	sqlite3_stmt *stmt = NULL;
+
+	ret = sqlite3_prepare_v2(dbh, "SELECT count(*) FROM clients WHERE "
+				      "id==?", -1, &stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: unable to prepare update statement: %s",
+				__func__, sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	ret = sqlite3_bind_blob(stmt, 1, (const void *)clname, namelen,
+				SQLITE_STATIC);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: bind blob failed: %s",
+				__func__, sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	ret = sqlite3_step(stmt);
+	if (ret != SQLITE_ROW) {
+		xlog(L_ERROR, "%s: unexpected return code from select: %d",
+				__func__, ret);
+		goto out_err;
+	}
+
+	ret = sqlite3_column_int(stmt, 0);
+	xlog(D_GENERAL, "%s: select returned %d rows", ret);
+	if (ret != 1) {
+		ret = -EACCES;
+		goto out_err;
+	}
+
+	sqlite3_finalize(stmt);
+	stmt = NULL;
+	ret = sqlite3_prepare_v2(dbh, "UPDATE OR FAIL clients SET "
+				      "time=strftime('%s', 'now') WHERE id==?",
+				 -1, &stmt, NULL);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: unable to prepare update statement: %s",
+				__func__, sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	ret = sqlite3_bind_blob(stmt, 1, (const void *)clname, namelen,
+				SQLITE_STATIC);
+	if (ret != SQLITE_OK) {
+		xlog(L_ERROR, "%s: bind blob failed: %s",
+				__func__, sqlite3_errmsg(dbh));
+		goto out_err;
+	}
+
+	ret = sqlite3_step(stmt);
+	if (ret == SQLITE_DONE)
+		ret = SQLITE_OK;
+	else
+		xlog(L_ERROR, "%s: unexpected return code from update: %s",
+				__func__, sqlite3_errmsg(dbh));
+
+out_err:
+	xlog(D_GENERAL, "%s: returning %d", __func__, ret);
+	sqlite3_finalize(stmt);
+	return ret;
+}
diff --git a/utils/nfsdcld/sqlite.h b/utils/nfsdcld/sqlite.h
index 425f5ba..59ebd72 100644
--- a/utils/nfsdcld/sqlite.h
+++ b/utils/nfsdcld/sqlite.h
@@ -23,5 +23,6 @@
 int sqlite_maindb_init(char *topdir);
 int sqlite_insert_client(const unsigned char *clname, const size_t namelen);
 int sqlite_remove_client(const unsigned char *clname, const size_t namelen);
+int sqlite_check_client(const unsigned char *clname, const size_t namelen);
 
 #endif /* _SQLITE_H */
-- 
1.7.7.6

[PATCH v5 6/9] nfsdcld: add function to remove unreclaimed client records

[Jeff Layton]

This should remove any client record that has a timestamp prior to
the given time.

Eventually, this call will need to be made cluster aware when this is
run in a clustered configuration. For now, this is only suitable for
single-host configurations.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 utils/nfsdcld/nfsdcld.c |   34 ++++++++++++++++++++++++++++++++++
 utils/nfsdcld/sqlite.c  |   28 ++++++++++++++++++++++++++++
 utils/nfsdcld/sqlite.h  |    1 +
 3 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/utils/nfsdcld/nfsdcld.c b/utils/nfsdcld/nfsdcld.c
index eb01d23..d2a5f2c 100644
--- a/utils/nfsdcld/nfsdcld.c
+++ b/utils/nfsdcld/nfsdcld.c
@@ -234,6 +234,37 @@ cld_check(struct cld_client *clnt)
 }
 
 static void
+cld_gracedone(struct cld_client *clnt)
+{
+	int ret;
+	ssize_t bsize, wsize;
+	struct cld_msg *cmsg = &clnt->cl_msg;
+
+	xlog(D_GENERAL, "%s: grace done. cm_gracetime=%ld", __func__,
+			cmsg->cm_u.cm_gracetime);
+
+	ret = sqlite_remove_unreclaimed(cmsg->cm_u.cm_gracetime);
+
+	/* set up reply: downcall with 0 status */
+	cmsg->cm_status = ret ? -EREMOTEIO : ret;
+
+	bsize = sizeof(*cmsg);
+
+	xlog(D_GENERAL, "Doing downcall with status %d", cmsg->cm_status);
+	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
+	if (wsize != bsize) {
+		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
+			 __func__, wsize);
+		ret = cld_pipe_open(clnt);
+		if (ret) {
+			xlog(L_FATAL, "%s: unable to reopen pipe: %d",
+					__func__, ret);
+			exit(ret);
+		}
+	}
+}
+
+static void
 cldcb(int UNUSED(fd), short which, void *data)
 {
 	ssize_t len;
@@ -267,6 +298,9 @@ cldcb(int UNUSED(fd), short which, void *data)
 	case Cld_Check:
 		cld_check(clnt);
 		break;
+	case Cld_GraceDone:
+		cld_gracedone(clnt);
+		break;
 	default:
 		xlog(L_WARNING, "%s: command %u is not yet implemented",
 				__func__, cmsg->cm_cmd);
diff --git a/utils/nfsdcld/sqlite.c b/utils/nfsdcld/sqlite.c
index 01bba1a..9e35774 100644
--- a/utils/nfsdcld/sqlite.c
+++ b/utils/nfsdcld/sqlite.c
@@ -38,6 +38,7 @@
 #include "config.h"
 #endif /* HAVE_CONFIG_H */
 
+#include <dirent.h>
 #include <errno.h>
 #include <event.h>
 #include <stdbool.h>
@@ -360,3 +361,30 @@ out_err:
 	sqlite3_finalize(stmt);
 	return ret;
 }
+
+/*
+ * remove any client records that were not reclaimed since grace_start.
+ */
+int
+sqlite_remove_unreclaimed(time_t grace_start)
+{
+	int ret;
+	char *err = NULL;
+
+	ret = snprintf(buf, sizeof(buf), "DELETE FROM clients WHERE time < %ld",
+			grace_start);
+	if (ret < 0) {
+		return ret;
+	} else if ((size_t)ret >= sizeof(buf)) {
+		ret = -EINVAL;
+		return ret;
+	}
+
+	ret = sqlite3_exec(dbh, buf, NULL, NULL, &err);
+	if (ret != SQLITE_OK)
+		xlog(L_ERROR, "%s: delete failed: %s", __func__, err);
+
+	xlog(D_GENERAL, "%s: returning %d", __func__, ret);
+	sqlite3_free(err);
+	return ret;
+}
diff --git a/utils/nfsdcld/sqlite.h b/utils/nfsdcld/sqlite.h
index 59ebd72..c85e7d6 100644
--- a/utils/nfsdcld/sqlite.h
+++ b/utils/nfsdcld/sqlite.h
@@ -24,5 +24,6 @@ int sqlite_maindb_init(char *topdir);
 int sqlite_insert_client(const unsigned char *clname, const size_t namelen);
 int sqlite_remove_client(const unsigned char *clname, const size_t namelen);
 int sqlite_check_client(const unsigned char *clname, const size_t namelen);
+int sqlite_remove_unreclaimed(const time_t grace_start);
 
 #endif /* _SQLITE_H */
-- 
1.7.7.6

[PATCH v5 7/9] nfsdcld: make it watch for inotify events in the containing directory

[Jeff Layton]

Before opening the pipe, set an inotify watch on the containing dir and
then try to open the pipe. If it succeeds, then set up pipe and inotify
events and return success. If it fails with -ENOENT, then just set up
the inotify event and return success. If it fails with any other error
then return the error and the caller can then abort the program.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 configure.ac            |    2 +-
 utils/nfsdcld/nfsdcld.c |  146 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 143 insertions(+), 5 deletions(-)

diff --git a/configure.ac b/configure.ac
index 134b609..191a72a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -270,7 +270,7 @@ if test "$enable_nfsv4" = yes; then
   AC_SQLITE3_VERS
 
   if test "$enable_nfsdcld" = "yes"; then
-    AC_CHECK_HEADERS([linux/nfsd/cld.h], ,
+    AC_CHECK_HEADERS([linux/nfsd/cld.h libgen.h sys/inotify.h], ,
 		     AC_MSG_ERROR([Cannot find header needed for nfsdcld]))
 
     if test "$libsqlite3_cv_is_recent" != "yes" ; then
diff --git a/utils/nfsdcld/nfsdcld.c b/utils/nfsdcld/nfsdcld.c
index d2a5f2c..9e7a9a5 100644
--- a/utils/nfsdcld/nfsdcld.c
+++ b/utils/nfsdcld/nfsdcld.c
@@ -32,6 +32,8 @@
 #include <sys/types.h>
 #include <fcntl.h>
 #include <unistd.h>
+#include <libgen.h>
+#include <sys/inotify.h>
 #include <linux/nfsd/cld.h>
 
 #include "xlog.h"
@@ -55,6 +57,8 @@ struct cld_client {
 
 /* global variables */
 static char *pipepath = DEFAULT_CLD_PATH;
+static int 		inotify_fd = -1;
+static struct event	pipedir_event;
 
 static struct option longopts[] =
 {
@@ -76,6 +80,8 @@ usage(char *progname)
 	printf("%s [ -hFd ] [ -p pipe ] [ -s dir ]\n", progname);
 }
 
+#define INOTIFY_EVENT_MAX (sizeof(struct inotify_event) + NAME_MAX)
+
 static int
 cld_pipe_open(struct cld_client *clnt)
 {
@@ -99,18 +105,149 @@ cld_pipe_open(struct cld_client *clnt)
 	return 0;
 }
 
+static void
+cld_inotify_cb(int UNUSED(fd), short which, void *data)
+{
+	int ret;
+	size_t elen;
+	ssize_t rret;
+	char evbuf[INOTIFY_EVENT_MAX];
+	char *dirc = NULL, *pname;
+	struct inotify_event *event = (struct inotify_event *)evbuf;
+	struct cld_client *clnt = data;
+
+	if (which != EV_READ)
+		return;
+
+	xlog(D_GENERAL, "%s: called for EV_READ", __func__);
+
+	dirc = strndup(pipepath, PATH_MAX);
+	if (!dirc) {
+		xlog(L_ERROR, "%s: unable to allocate memory", __func__);
+		goto out;
+	}
+
+	rret = read(inotify_fd, evbuf, INOTIFY_EVENT_MAX);
+	if (rret < 0) {
+		xlog(L_ERROR, "%s: read from inotify fd failed: %m", __func__);
+		goto out;
+	}
+
+	/* check to see if we have a filename in the evbuf */
+	if (!event->len) {
+		xlog(D_GENERAL, "%s: no filename in inotify event", __func__);
+		goto out;
+	}
+
+	pname = basename(dirc);
+	elen = strnlen(event->name, event->len);
+
+	/* does the filename match our pipe? */
+	if (strlen(pname) != elen || memcmp(pname, event->name, elen)) {
+		xlog(D_GENERAL, "%s: wrong filename (%s)", __func__,
+				event->name);
+		goto out;
+	}
+
+	ret = cld_pipe_open(clnt);
+	switch (ret) {
+	case 0:
+		/* readd the event for the cl_event pipe */
+		event_add(&clnt->cl_event, NULL);
+		break;
+	case -ENOENT:
+		/* pipe must have disappeared, wait for it to come back */
+		goto out;
+	default:
+		/* anything else is fatal */
+		xlog(L_FATAL, "%s: unable to open new pipe (%d). Aborting.",
+			ret, __func__);
+		exit(ret);
+	}
+
+out:
+	event_add(&pipedir_event, NULL);
+	free(dirc);
+}
+
+static int
+cld_inotify_setup(void)
+{
+	int ret;
+	char *dirc, *dname;
+
+	dirc = strndup(pipepath, PATH_MAX);
+	if (!dirc) {
+		xlog_err("%s: unable to allocate memory", __func__);
+		ret = -ENOMEM;
+		goto out_free;
+	}
+
+	dname = dirname(dirc);
+
+	inotify_fd = inotify_init();
+	if (inotify_fd < 0) {
+		xlog_err("%s: inotify_init failed: %m", __func__);
+		ret = -errno;
+		goto out_free;
+	}
+
+	ret = inotify_add_watch(inotify_fd, dname, IN_CREATE);
+	if (ret < 0) {
+		xlog_err("%s: inotify_add_watch failed: %m", __func__);
+		ret = -errno;
+		goto out_err;
+	}
+
+out_free:
+	free(dirc);
+	return 0;
+out_err:
+	close(inotify_fd);
+	goto out_free;
+}
+
+/*
+ * Set an inotify watch on the directory that should contain the pipe, and then
+ * try to open it. If it fails with anything but -ENOENT, return the error
+ * immediately.
+ *
+ * If it succeeds, then set up the pipe event handler. At that point, set up
+ * the inotify event handler and go ahead and return success.
+ */
 static int
 cld_pipe_init(struct cld_client *clnt)
 {
 	int ret;
 
+	xlog(D_GENERAL, "%s: init pipe handlers", __func__);
+
+	ret = cld_inotify_setup();
+	if (ret != 0)
+		goto out;
+
 	clnt->cl_fd = -1;
 	ret = cld_pipe_open(clnt);
-	if (ret)
-		return ret;
+	switch (ret) {
+	case 0:
+		/* add the event and we're good to go */
+		event_add(&clnt->cl_event, NULL);
+		break;
+	case -ENOENT:
+		/* ignore this error -- cld_inotify_cb will handle it */
+		ret = 0;
+		break;
+	default:
+		/* anything else is fatal */
+		close(inotify_fd);
+		goto out;
+	}
 
-	event_add(&clnt->cl_event, NULL);
-	return 0;
+	/* set event for inotify read */
+	event_set(&pipedir_event, inotify_fd, EV_READ, cld_inotify_cb, clnt);
+	event_add(&pipedir_event, NULL);
+out:
+	return ret;
 }
 
 static void
@@ -384,6 +521,7 @@ main(int argc, char **argv)
 		xlog(L_ERROR, "%s: event_dispatch failed: %m", __func__);
 
 	close(clnt.cl_fd);
+	close(inotify_fd);
 out:
 	free(progname);
 	return rc;
-- 
1.7.7.6

[PATCH v5 8/9] nfsdcld: add a manpage for nfsdcld

[Jeff Layton]

...for now I'm including the POD source for the manpage. We can drop
from the repo if desired though.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 utils/nfsdcld/Makefile.am |    4 +-
 utils/nfsdcld/nfsdcld.man |  180 +++++++++++++++++++++++++++++++++++++++++++++
 utils/nfsdcld/nfsdcld.pod |   67 +++++++++++++++++
 3 files changed, 249 insertions(+), 2 deletions(-)
 create mode 100644 utils/nfsdcld/nfsdcld.man
 create mode 100644 utils/nfsdcld/nfsdcld.pod

diff --git a/utils/nfsdcld/Makefile.am b/utils/nfsdcld/Makefile.am
index 8e4f2ab..f320dff 100644
--- a/utils/nfsdcld/Makefile.am
+++ b/utils/nfsdcld/Makefile.am
@@ -1,7 +1,7 @@
 ## Process this file with automake to produce Makefile.in
 
-#man8_MANS	= nfsdcld.man
-#EXTRA_DIST = $(man8_MANS)
+man8_MANS	= nfsdcld.man
+EXTRA_DIST	= $(man8_MANS)
 
 AM_CFLAGS	+= -D_LARGEFILE64_SOURCE
 sbin_PROGRAMS	= nfsdcld
diff --git a/utils/nfsdcld/nfsdcld.man b/utils/nfsdcld/nfsdcld.man
new file mode 100644
index 0000000..bad5f34
--- /dev/null
+++ b/utils/nfsdcld/nfsdcld.man
@@ -0,0 +1,180 @@
+.\" Automatically generated by Pod::Man 2.22 (Pod::Simple 3.13)
+.\"
+.\" Standard preamble:
+.\" ========================================================================
+.de Sp \" Vertical space (when we can't use .PP)
+.if t .sp .5v
+.if n .sp
+..
+.de Vb \" Begin verbatim text
+.ft CW
+.nf
+.ne \\$1
+..
+.de Ve \" End verbatim text
+.ft R
+.fi
+..
+.\" Set up some character translations and predefined strings.  \*(-- will
+.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
+.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
+.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
+.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
+.\" nothing in troff, for use with C<>.
+.tr \(*W-
+.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
+.ie n \{\
+.    ds -- \(*W-
+.    ds PI pi
+.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
+.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
+.    ds L" ""
+.    ds R" ""
+.    ds C` ""
+.    ds C' ""
+'br\}
+.el\{\
+.    ds -- \|\(em\|
+.    ds PI \(*p
+.    ds L" ``
+.    ds R" ''
+'br\}
+.\"
+.\" Escape single quotes in literal strings from groff's Unicode transform.
+.ie \n(.g .ds Aq \(aq
+.el       .ds Aq '
+.\"
+.\" If the F register is turned on, we'll generate index entries on stderr for
+.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
+.\" entries marked with X<> in POD.  Of course, you'll have to process the
+.\" output yourself in some meaningful fashion.
+.ie \nF \{\
+.    de IX
+.    tm Index:\\$1\t\\n%\t"\\$2"
+..
+.    nr % 0
+.    rr F
+.\}
+.el \{\
+.    de IX
+..
+.\}
+.\"
+.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
+.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
+.    \" fudge factors for nroff and troff
+.if n \{\
+.    ds #H 0
+.    ds #V .8m
+.    ds #F .3m
+.    ds #[ \f1
+.    ds #] \fP
+.\}
+.if t \{\
+.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
+.    ds #V .6m
+.    ds #F 0
+.    ds #[ \&
+.    ds #] \&
+.\}
+.    \" simple accents for nroff and troff
+.if n \{\
+.    ds ' \&
+.    ds ` \&
+.    ds ^ \&
+.    ds , \&
+.    ds ~ ~
+.    ds /
+.\}
+.if t \{\
+.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
+.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
+.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
+.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
+.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
+.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
+.\}
+.    \" troff and (daisy-wheel) nroff accents
+.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
+.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
+.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
+.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
+.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
+.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
+.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
+.ds ae a\h'-(\w'a'u*4/10)'e
+.ds Ae A\h'-(\w'A'u*4/10)'E
+.    \" corrections for vroff
+.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
+.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
+.    \" for low resolution devices (crt and lpr)
+.if \n(.H>23 .if \n(.V>19 \
+\{\
+.    ds : e
+.    ds 8 ss
+.    ds o a
+.    ds d- d\h'-1'\(ga
+.    ds D- D\h'-1'\(hy
+.    ds th \o'bp'
+.    ds Th \o'LP'
+.    ds ae ae
+.    ds Ae AE
+.\}
+.rm #[ #] #H #V #F C
+.\" ========================================================================
+.\"
+.IX Title "NFSDCLD 8"
+.TH NFSDCLD 8 "2011-12-21" "" ""
+.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
+.\" way too many mistakes in technical documents.
+.if n .ad l
+.nh
+.SH "NAME"
+nfsdcld \- NFSv4 Client Tracking Daemon
+.SH "SYNOPSIS"
+.IX Header "SYNOPSIS"
+nfsdcld [\-d] [\-F] [\-p path] [\-s stable storage dir]
+.SH "DESCRIPTION"
+.IX Header "DESCRIPTION"
+nfsdcld is the NFSv4 client tracking daemon. It is not necessary to run
+this daemon on machines that are not acting as NFSv4 servers.
+.PP
+When a network partition is combined with a server reboot, there are
+edge conditions that can cause the server to grant lock reclaims when
+other clients have taken conflicting locks in the interim. A more detailed
+explanation of this issue is described in \s-1RFC\s0 3530, section 8.6.3.
+.PP
+In order to prevent these problems, the server must track a small amount
+of per-client information on stable storage. This daemon provides the
+userspace piece of that functionality.
+.SH "OPTIONS"
+.IX Header "OPTIONS"
+.IP "\fB\-d\fR, \fB\-\-debug\fR" 4
+.IX Item "-d, --debug"
+Enable debug level logging.
+.IP "\fB\-F\fR, \fB\-\-foreground\fR" 4
+.IX Item "-F, --foreground"
+Runs the daemon in the foreground and prints all output to stderr
+.IP "\fB\-p\fR \fIpipe\fR, \fB\-\-pipe\fR=\fIpipe\fR" 4
+.IX Item "-p pipe, --pipe=pipe"
+Location of the \*(L"cld\*(R" upcall pipe. The default value is
+\&\fI/var/lib/nfs/rpc_pipefs/nfsd/cld\fR. If the pipe does not exist when the
+daemon starts then it will wait for it to be created.
+.IP "\fB\-s\fR \fIstoragedir\fR, \fB\-\-storagedir\fR=\fIstorage_dir\fR" 4
+.IX Item "-s storagedir, --storagedir=storage_dir"
+Directory where stable storage information should be kept. The default
+value is \fI/var/lib/nfs/nfsdcld\fR.
+.SH "NOTES"
+.IX Header "NOTES"
+The Linux kernel NFSv4 server has historically tracked this information
+on stable storage by manipulating information on the filesystem
+directly, in the directory to which \fI/proc/fs/nfsd/nfsv4recoverydir\fR
+points.
+.PP
+This daemon requires a kernel that supports the nfsdcld upcall. If the
+kernel does not support the new upcall, or is using the legacy client
+name tracking code then it will not create the pipe that nfsdcld uses to
+talk to the kernel.
+.SH "AUTHORS"
+.IX Header "AUTHORS"
+The nfsdcld daemon was developed by Jeff Layton <jlayton@redhat.com>.
diff --git a/utils/nfsdcld/nfsdcld.pod b/utils/nfsdcld/nfsdcld.pod
new file mode 100644
index 0000000..2ff1d3d
--- /dev/null
+++ b/utils/nfsdcld/nfsdcld.pod
@@ -0,0 +1,67 @@
+# turn into a manpage with the following command:
+#
+# pod2man -s 8 -c '' -r '' --stderr nfsdcld.pod > nfsdcld.man
+#
+
+=head1 NAME
+
+nfsdcld - NFSv4 Client Tracking Daemon
+
+=head1 SYNOPSIS
+
+nfsdcld [-d] [-F] [-p path] [-s stable storage dir]
+
+=head1 DESCRIPTION
+
+nfsdcld is the NFSv4 client tracking daemon. It is not necessary to run
+this daemon on machines that are not acting as NFSv4 servers.
+
+When a network partition is combined with a server reboot, there are
+edge conditions that can cause the server to grant lock reclaims when
+other clients have taken conflicting locks in the interim. A more detailed
+explanation of this issue is described in RFC 3530, section 8.6.3.
+
+In order to prevent these problems, the server must track a small amount
+of per-client information on stable storage. This daemon provides the
+userspace piece of that functionality.
+
+=head1 OPTIONS
+
+=over
+
+=item B<-d>, B<--debug>
+
+Enable debug level logging.
+
+=item B<-F>, B<--foreground>
+
+Runs the daemon in the foreground and prints all output to stderr
+
+=item B<-p> I<pipe>, B<--pipe>=I<pipe>
+
+Location of the "cld" upcall pipe. The default value is
+I</var/lib/nfs/rpc_pipefs/nfsd/cld>. If the pipe does not exist when the
+daemon starts then it will wait for it to be created.
+
+=item B<-s> I<storagedir>, B<--storagedir>=I<storage_dir>
+
+Directory where stable storage information should be kept. The default
+value is I</var/lib/nfs/nfsdcld>.
+
+=back
+
+=head1 NOTES
+
+The Linux kernel NFSv4 server has historically tracked this information
+on stable storage by manipulating information on the filesystem
+directly, in the directory to which I</proc/fs/nfsd/nfsv4recoverydir>
+points.
+
+This daemon requires a kernel that supports the nfsdcld upcall. If the
+kernel does not support the new upcall, or is using the legacy client
+name tracking code then it will not create the pipe that nfsdcld uses to
+talk to the kernel.
+
+=head1 AUTHORS
+
+The nfsdcld daemon was developed by Jeff Layton <jlayton@redhat.com>.
-- 
1.7.7.6

[PATCH v5 9/9] nfsdcld: update the README

[Jeff Layton]

Add info to the README about when this daemon should be started, and
its build and runtime requirements.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
---
 README |   25 +++++++++++++++++++++++--
 1 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/README b/README
index e7588cf..cb1d2f5 100644
--- a/README
+++ b/README
@@ -15,6 +15,8 @@ libraries.  They are available from
    http://www.citi.umich.edu/projects/nfsv4/linux/libnfsidmap/
 Otherwise use --disable-nfsv4
 
+To use the nfsdcld tracking daemon, nfsv4 support must be enabled,
+and the libsqlite3 development libraries must be installed.
 
 1. COMPILING
 
@@ -106,12 +108,31 @@ scripts can be written to work correctly.
        the lock.
        rpc.statd is only needed for NFSv2 and NFSv3 support.
 
-   E/ rpc.nfsd
+   E/ nfsdcld
+       This daemon is only needed on kernels that support the nfsdcld
+       upcall, and only if the legacy client ID tracking isn't used. It
+       is also not needed if the server does not support NFSv4.
+
+       To determine whether you need this or not, do the following:
+
+           # cat /proc/fs/nfsd/versions
+
+       That should yield a list of NFS versions that this kernel supports,
+       if "4" or later is not in that list, or they are prefixed with a "-"
+       then you don't need to run this daemon. Next:
+
+           # cat /proc/fs/nfsd/nfsv4recoverydir
+
+       If that file is not present, or the directory that the above command
+       outputs is not present, then this daemon is required in order to
+       support lock recovery by the clients when the server reboots.
+
+   F/ rpc.nfsd
        Starting nfsd will automatically start lockd.  The nfs server
        will now be fully active and respond to any requests from
        clients.
        
-   F/ sm-notify
+   G/ sm-notify
        This will notify any client which might have locks from before
        a reboot to try to reclaim their locks.  This should start
        immediately after rpc.nfsd is started so that clients have a
-- 
1.7.7.6

Re: [PATCH v5 1/9] nfsdcld: add client tracking daemon stub

[Steve Dickson]

On 02/01/2012 10:44 AM, Jeff Layton wrote:
> This program opens and "listens" on the new nfsd/cld rpc_pipefs pipe.
> The code here doesn't actually do anything on stable storage yet. That
> will be added in a later patch.
> 
> The patch also adds a autoconf enable switch for the new daemon that
> defaults to "no", and a test for the upcall description header file.
> 
> Signed-off-by: Jeff Layton <jlayton@redhat.com>
> ---
>  configure.ac              |   14 +++
>  utils/Makefile.am         |    4 +
>  utils/nfsdcld/Makefile.am |   14 +++
>  utils/nfsdcld/nfsdcld.c   |  275 +++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 307 insertions(+), 0 deletions(-)
>  create mode 100644 utils/nfsdcld/Makefile.am
>  create mode 100644 utils/nfsdcld/nfsdcld.c
> 
> diff --git a/configure.ac b/configure.ac
> index 920e8da..d50e54e 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -185,6 +185,12 @@ else
>  	AM_CONDITIONAL(MOUNT_CONFIG, [test "$enable_mount" = "yes"])
>  fi
>  
> +AC_ARG_ENABLE(nfsdcld,
> +	[AC_HELP_STRING([--enable-nfsdcld],
> +			[Create nfsdcld NFSv4 clientid tracking daemon. <:@default=no@:>@])],
> +	enable_nfsdcld=$enableval,
> +	enable_nfsdcld="no")
> +
>  dnl Check for TI-RPC library and headers
>  AC_LIBTIRPC
>  
> @@ -260,6 +266,13 @@ if test "$enable_nfsv4" = yes; then
>    dnl check for the keyutils libraries and headers
>    AC_KEYUTILS
>  
> +  if test "$enable_nfsdcld" = "yes"; then
> +    AC_CHECK_HEADERS([linux/nfsd/cld.h], ,
> +		     AC_MSG_ERROR([Cannot find header needed for nfsdcld]))
> +  fi
This adds a dependency on the kernel headers being present which 
is unprecedented for nfs-utils or any other package I'm aware of
and I just don't think its a good idea... 

So I'm wondering if there is someway around creating this
dependency...

steved.


> +
> +  AM_CONDITIONAL(CONFIG_NFSDCLD, [test "$enable_nfsdcld" = "yes" ])
> +
>    dnl librpcsecgss already has a dependency on libgssapi,
>    dnl but we need to make sure we get the right version
>    if test "$enable_gss" = yes; then
> @@ -460,6 +473,7 @@ AC_CONFIG_FILES([
>  	tools/nfs-iostat/Makefile
>  	utils/Makefile
>  	utils/blkmapd/Makefile
> +	utils/nfsdcld/Makefile
>  	utils/exportfs/Makefile
>  	utils/gssd/Makefile
>  	utils/idmapd/Makefile
> diff --git a/utils/Makefile.am b/utils/Makefile.am
> index d074b85..5df7ca7 100644
> --- a/utils/Makefile.am
> +++ b/utils/Makefile.am
> @@ -21,6 +21,10 @@ if CONFIG_MOUNT
>  OPTDIRS += mount
>  endif
>  
> +if CONFIG_NFSDCLD
> +OPTDIRS += nfsdcld
> +endif
> +
>  SUBDIRS = \
>  	exportfs \
>  	mountd \
> diff --git a/utils/nfsdcld/Makefile.am b/utils/nfsdcld/Makefile.am
> new file mode 100644
> index 0000000..ed7ed42
> --- /dev/null
> +++ b/utils/nfsdcld/Makefile.am
> @@ -0,0 +1,14 @@
> +## Process this file with automake to produce Makefile.in
> +
> +#man8_MANS	= nfsdcld.man
> +#EXTRA_DIST = $(man8_MANS)
> +
> +AM_CFLAGS	+= -D_LARGEFILE64_SOURCE
> +sbin_PROGRAMS	= nfsdcld
> +
> +nfsdcld_SOURCES = nfsdcld.c
> +
> +nfsdcld_LDADD = ../../support/nfs/libnfs.a $(LIBEVENT)
> +
> +MAINTAINERCLEANFILES = Makefile.in
> +
> diff --git a/utils/nfsdcld/nfsdcld.c b/utils/nfsdcld/nfsdcld.c
> new file mode 100644
> index 0000000..7a83fbe
> --- /dev/null
> +++ b/utils/nfsdcld/nfsdcld.c
> @@ -0,0 +1,275 @@
> +/*
> + * nfsdcld.c -- NFSv4 client name tracking daemon
> + *
> + * Copyright (C) 2011  Red Hat, Jeff Layton <jlayton@redhat.com>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version 2
> + * of the License, or (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor,
> + * Boston, MA 02110-1301, USA.
> + */
> +
> +#ifdef HAVE_CONFIG_H
> +#include "config.h"
> +#endif /* HAVE_CONFIG_H */
> +
> +#include <errno.h>
> +#include <event.h>
> +#include <stdbool.h>
> +#include <getopt.h>
> +#include <string.h>
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +#include <fcntl.h>
> +#include <unistd.h>
> +#include <linux/nfsd/cld.h>
> +
> +#include "xlog.h"
> +#include "nfslib.h"
> +
> +#ifndef PIPEFS_DIR
> +#define PIPEFS_DIR NFS_STATEDIR "/rpc_pipefs"
> +#endif
> +
> +#define DEFAULT_CLD_PATH	PIPEFS_DIR "/nfsd/cld"
> +
> +#define UPCALL_VERSION		1
> +
> +/* private data structures */
> +struct cld_client {
> +	int			cl_fd;
> +	struct event		cl_event;
> +	struct cld_msg	cl_msg;
> +};
> +
> +/* global variables */
> +static char *pipepath = DEFAULT_CLD_PATH;
> +
> +static struct option longopts[] =
> +{
> +	{ "help", 0, NULL, 'h' },
> +	{ "foreground", 0, NULL, 'F' },
> +	{ "debug", 0, NULL, 'd' },
> +	{ "pipe", 1, NULL, 'p' },
> +	{ NULL, 0, 0, 0 },
> +};
> +
> +/* forward declarations */
> +static void cldcb(int UNUSED(fd), short which, void *data);
> +
> +static void
> +usage(char *progname)
> +{
> +	printf("Usage:\n");
> +	printf("%s [ -hFd ] [ -p pipe ]\n", progname);
> +}
> +
> +static int
> +cld_pipe_open(struct cld_client *clnt)
> +{
> +	int fd;
> +
> +	xlog(D_GENERAL, "%s: opening upcall pipe %s", __func__, pipepath);
> +	fd = open(pipepath, O_RDWR, 0);
> +	if (fd < 0) {
> +		xlog(L_ERROR, "%s: open of %s failed: %m", __func__, pipepath);
> +		return -errno;
> +	}
> +
> +	if (clnt->cl_event.ev_flags & EVLIST_INIT)
> +		event_del(&clnt->cl_event);
> +	if (clnt->cl_fd >= 0)
> +		close(clnt->cl_fd);
> +
> +	clnt->cl_fd = fd;
> +	event_set(&clnt->cl_event, clnt->cl_fd, EV_READ, cldcb, clnt);
> +	/* event_add is done by the caller */
> +	return 0;
> +}
> +
> +static int
> +cld_pipe_init(struct cld_client *clnt)
> +{
> +	int ret;
> +
> +	clnt->cl_fd = -1;
> +	ret = cld_pipe_open(clnt);
> +	if (ret)
> +		return ret;
> +
> +	event_add(&clnt->cl_event, NULL);
> +	return 0;
> +}
> +
> +static void
> +cld_not_implemented(struct cld_client *clnt)
> +{
> +	int ret;
> +	ssize_t bsize, wsize;
> +	struct cld_msg *cmsg = &clnt->cl_msg;
> +
> +	xlog(D_GENERAL, "%s: downcalling with not implemented error", __func__);
> +
> +	/* set up reply */
> +	cmsg->cm_status = -EOPNOTSUPP;
> +
> +	bsize = sizeof(*cmsg);
> +
> +	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
> +	if (wsize != bsize)
> +		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
> +			 __func__, wsize);
> +
> +	/* reopen pipe, just to be sure */
> +	ret = cld_pipe_open(clnt);
> +	if (ret) {
> +		xlog(L_FATAL, "%s: unable to reopen pipe: %d", __func__, ret);
> +		exit(ret);
> +	}
> +}
> +
> +static void
> +cld_create(struct cld_client *clnt)
> +{
> +	int ret;
> +	ssize_t bsize, wsize;
> +	struct cld_msg *cmsg = &clnt->cl_msg;
> +
> +	xlog(D_GENERAL, "%s: create client record", __func__);
> +
> +	/* FIXME: create client record on storage here */
> +
> +	/* set up reply */
> +	cmsg->cm_status = 0;
> +
> +	bsize = sizeof(*cmsg);
> +
> +	wsize = atomicio((void *)write, clnt->cl_fd, cmsg, bsize);
> +	if (wsize != bsize) {
> +		xlog(L_ERROR, "%s: problem writing to cld pipe (%ld): %m",
> +			 __func__, wsize);
> +		ret = cld_pipe_open(clnt);
> +		if (ret) {
> +			xlog(L_FATAL, "%s: unable to reopen pipe: %d",
> +					__func__, ret);
> +			exit(ret);
> +		}
> +	}
> +}
> +
> +static void
> +cldcb(int UNUSED(fd), short which, void *data)
> +{
> +	ssize_t len;
> +	struct cld_client *clnt = data;
> +	struct cld_msg *cmsg = &clnt->cl_msg;
> +
> +	if (which != EV_READ)
> +		goto out;
> +
> +	len = atomicio(read, clnt->cl_fd, cmsg, sizeof(*cmsg));
> +	if (len <= 0) {
> +		xlog(L_ERROR, "%s: pipe read failed: %m", __func__);
> +		cld_pipe_open(clnt);
> +		goto out;
> +	}
> +
> +	if (cmsg->cm_vers != UPCALL_VERSION) {
> +		xlog(L_ERROR, "%s: unsupported upcall version: %hu",
> +				cmsg->cm_vers);
> +		cld_pipe_open(clnt);
> +		goto out;
> +	}
> +
> +	switch(cmsg->cm_cmd) {
> +	case Cld_Create:
> +		cld_create(clnt);
> +		break;
> +	default:
> +		xlog(L_WARNING, "%s: command %u is not yet implemented",
> +				__func__, cmsg->cm_cmd);
> +		cld_not_implemented(clnt);
> +	}
> +out:
> +	event_add(&clnt->cl_event, NULL);
> +}
> +
> +int
> +main(int argc, char **argv)
> +{
> +	char arg;
> +	int rc = 0;
> +	bool foreground = false;
> +	char *progname;
> +	struct cld_client clnt;
> +
> +	memset(&clnt, 0, sizeof(clnt));
> +
> +	progname = strdup(basename(argv[0]));
> +	if (!progname) {
> +		fprintf(stderr, "%s: unable to allocate memory.\n", argv[0]);
> +		return 1;
> +	}
> +
> +	event_init();
> +	xlog_syslog(0);
> +	xlog_stderr(1);
> +
> +	/* process command-line options */
> +	while ((arg = getopt_long(argc, argv, "hdFp:", longopts,
> +				  NULL)) != EOF) {
> +		switch (arg) {
> +		case 'd':
> +			xlog_config(D_ALL, 1);
> +			break;
> +		case 'F':
> +			foreground = true;
> +			break;
> +		case 'p':
> +			pipepath = optarg;
> +			break;
> +		default:
> +			usage(progname);
> +			return 0;
> +		}
> +	}
> +
> +
> +	xlog_open(progname);
> +	if (!foreground) {
> +		xlog_syslog(1);
> +		xlog_stderr(0);
> +		rc = daemon(0, 0);
> +		if (rc) {
> +			xlog(L_ERROR, "Unable to daemonize: %m");
> +			goto out;
> +		}
> +	}
> +
> +	/* set up storage db */
> +
> +	/* set up event handler */
> +	rc = cld_pipe_init(&clnt);
> +	if (rc)
> +		goto out;
> +
> +	xlog(D_GENERAL, "%s: Starting event dispatch handler.", __func__);
> +	rc = event_dispatch();
> +	if (rc < 0)
> +		xlog(L_ERROR, "%s: event_dispatch failed: %m", __func__);
> +
> +	close(clnt.cl_fd);
> +out:
> +	free(progname);
> +	return rc;
> +}

Re: [PATCH v5 1/9] nfsdcld: add client tracking daemon stub

[Jeff Layton]

On Mon, 16 Apr 2012 10:06:55 -0400
Steve Dickson <SteveD@redhat.com> wrote:

> 
> 
> On 02/01/2012 10:44 AM, Jeff Layton wrote:
> > This program opens and "listens" on the new nfsd/cld rpc_pipefs pipe.
> > The code here doesn't actually do anything on stable storage yet. That
> > will be added in a later patch.
> > 
> > The patch also adds a autoconf enable switch for the new daemon that
> > defaults to "no", and a test for the upcall description header file.
> > 
> > Signed-off-by: Jeff Layton <jlayton@redhat.com>
> > ---
> >  configure.ac              |   14 +++
> >  utils/Makefile.am         |    4 +
> >  utils/nfsdcld/Makefile.am |   14 +++
> >  utils/nfsdcld/nfsdcld.c   |  275 +++++++++++++++++++++++++++++++++++++++++++++
> >  4 files changed, 307 insertions(+), 0 deletions(-)
> >  create mode 100644 utils/nfsdcld/Makefile.am
> >  create mode 100644 utils/nfsdcld/nfsdcld.c
> > 
> > diff --git a/configure.ac b/configure.ac
> > index 920e8da..d50e54e 100644
> > --- a/configure.ac
> > +++ b/configure.ac
> > @@ -185,6 +185,12 @@ else
> >  	AM_CONDITIONAL(MOUNT_CONFIG, [test "$enable_mount" = "yes"])
> >  fi
> >  
> > +AC_ARG_ENABLE(nfsdcld,
> > +	[AC_HELP_STRING([--enable-nfsdcld],
> > +			[Create nfsdcld NFSv4 clientid tracking daemon. <:@default=no@:>@])],
> > +	enable_nfsdcld=$enableval,
> > +	enable_nfsdcld="no")
> > +
> >  dnl Check for TI-RPC library and headers
> >  AC_LIBTIRPC
> >  
> > @@ -260,6 +266,13 @@ if test "$enable_nfsv4" = yes; then
> >    dnl check for the keyutils libraries and headers
> >    AC_KEYUTILS
> >  
> > +  if test "$enable_nfsdcld" = "yes"; then
> > +    AC_CHECK_HEADERS([linux/nfsd/cld.h], ,
> > +		     AC_MSG_ERROR([Cannot find header needed for nfsdcld]))
> > +  fi
> This adds a dependency on the kernel headers being present which 
> is unprecedented for nfs-utils or any other package I'm aware of
> and I just don't think its a good idea... 
> 
> So I'm wondering if there is someway around creating this
> dependency...
> 

The idea with that header was to have a common userspace/kernel header
that describes the upcall/downcall format. The proper place for those
is generally the kernel-headers package (at least on Fedora). Other
distros have a similar package.

One possibility is to include a copy of this header with nfs-utils, and
only use it if /usr/include/linux/nfsd/cld.h isn't present. It's not
ideal, but I suppose we could live with it if adding a dependency on
kernel headers is anathema. It would take some rework of the code and
autoconf/automake stuff however.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH v5 1/9] nfsdcld: add client tracking daemon stub

[Steve Dickson]

On 04/16/2012 10:23 AM, Jeff Layton wrote:
> On Mon, 16 Apr 2012 10:06:55 -0400
> Steve Dickson <SteveD@redhat.com> wrote:
> 
>>
>>
>> On 02/01/2012 10:44 AM, Jeff Layton wrote:
>>> This program opens and "listens" on the new nfsd/cld rpc_pipefs pipe.
>>> The code here doesn't actually do anything on stable storage yet. That
>>> will be added in a later patch.
>>>
>>> The patch also adds a autoconf enable switch for the new daemon that
>>> defaults to "no", and a test for the upcall description header file.
>>>
>>> Signed-off-by: Jeff Layton <jlayton@redhat.com>
>>> ---
>>>  configure.ac              |   14 +++
>>>  utils/Makefile.am         |    4 +
>>>  utils/nfsdcld/Makefile.am |   14 +++
>>>  utils/nfsdcld/nfsdcld.c   |  275 +++++++++++++++++++++++++++++++++++++++++++++
>>>  4 files changed, 307 insertions(+), 0 deletions(-)
>>>  create mode 100644 utils/nfsdcld/Makefile.am
>>>  create mode 100644 utils/nfsdcld/nfsdcld.c
>>>
>>> diff --git a/configure.ac b/configure.ac
>>> index 920e8da..d50e54e 100644
>>> --- a/configure.ac
>>> +++ b/configure.ac
>>> @@ -185,6 +185,12 @@ else
>>>  	AM_CONDITIONAL(MOUNT_CONFIG, [test "$enable_mount" = "yes"])
>>>  fi
>>>  
>>> +AC_ARG_ENABLE(nfsdcld,
>>> +	[AC_HELP_STRING([--enable-nfsdcld],
>>> +			[Create nfsdcld NFSv4 clientid tracking daemon. <:@default=no@:>@])],
>>> +	enable_nfsdcld=$enableval,
>>> +	enable_nfsdcld="no")
>>> +
>>>  dnl Check for TI-RPC library and headers
>>>  AC_LIBTIRPC
>>>  
>>> @@ -260,6 +266,13 @@ if test "$enable_nfsv4" = yes; then
>>>    dnl check for the keyutils libraries and headers
>>>    AC_KEYUTILS
>>>  
>>> +  if test "$enable_nfsdcld" = "yes"; then
>>> +    AC_CHECK_HEADERS([linux/nfsd/cld.h], ,
>>> +		     AC_MSG_ERROR([Cannot find header needed for nfsdcld]))
>>> +  fi
>> This adds a dependency on the kernel headers being present which 
>> is unprecedented for nfs-utils or any other package I'm aware of
>> and I just don't think its a good idea... 
>>
>> So I'm wondering if there is someway around creating this
>> dependency...
>>
> 
> The idea with that header was to have a common userspace/kernel header
> that describes the upcall/downcall format. The proper place for those
> is generally the kernel-headers package (at least on Fedora). Other
> distros have a similar package.
> 
> One possibility is to include a copy of this header with nfs-utils, and
> only use it if /usr/include/linux/nfsd/cld.h isn't present. It's not
> ideal, but I suppose we could live with it if adding a dependency on
> kernel headers is anathema. It would take some rework of the code and
> autoconf/automake stuff however.
> 
I believe this is how it was done in the past, keeping separate
header files... Since I'm causing the pain, I would be more than 
willing to do the autoconf/automake updates...

steved.

Re: [PATCH v5 1/9] nfsdcld: add client tracking daemon stub

[Jeff Layton]

On Mon, 16 Apr 2012 11:14:34 -0400
Steve Dickson <SteveD@redhat.com> wrote:

> 
> 
> On 04/16/2012 10:23 AM, Jeff Layton wrote:
> > On Mon, 16 Apr 2012 10:06:55 -0400
> > Steve Dickson <SteveD@redhat.com> wrote:
> > 
> >>
> >>
> >> On 02/01/2012 10:44 AM, Jeff Layton wrote:
> >>> This program opens and "listens" on the new nfsd/cld rpc_pipefs pipe.
> >>> The code here doesn't actually do anything on stable storage yet. That
> >>> will be added in a later patch.
> >>>
> >>> The patch also adds a autoconf enable switch for the new daemon that
> >>> defaults to "no", and a test for the upcall description header file.
> >>>
> >>> Signed-off-by: Jeff Layton <jlayton@redhat.com>
> >>> ---
> >>>  configure.ac              |   14 +++
> >>>  utils/Makefile.am         |    4 +
> >>>  utils/nfsdcld/Makefile.am |   14 +++
> >>>  utils/nfsdcld/nfsdcld.c   |  275 +++++++++++++++++++++++++++++++++++++++++++++
> >>>  4 files changed, 307 insertions(+), 0 deletions(-)
> >>>  create mode 100644 utils/nfsdcld/Makefile.am
> >>>  create mode 100644 utils/nfsdcld/nfsdcld.c
> >>>
> >>> diff --git a/configure.ac b/configure.ac
> >>> index 920e8da..d50e54e 100644
> >>> --- a/configure.ac
> >>> +++ b/configure.ac
> >>> @@ -185,6 +185,12 @@ else
> >>>  	AM_CONDITIONAL(MOUNT_CONFIG, [test "$enable_mount" = "yes"])
> >>>  fi
> >>>  
> >>> +AC_ARG_ENABLE(nfsdcld,
> >>> +	[AC_HELP_STRING([--enable-nfsdcld],
> >>> +			[Create nfsdcld NFSv4 clientid tracking daemon. <:@default=no@:>@])],
> >>> +	enable_nfsdcld=$enableval,
> >>> +	enable_nfsdcld="no")
> >>> +
> >>>  dnl Check for TI-RPC library and headers
> >>>  AC_LIBTIRPC
> >>>  
> >>> @@ -260,6 +266,13 @@ if test "$enable_nfsv4" = yes; then
> >>>    dnl check for the keyutils libraries and headers
> >>>    AC_KEYUTILS
> >>>  
> >>> +  if test "$enable_nfsdcld" = "yes"; then
> >>> +    AC_CHECK_HEADERS([linux/nfsd/cld.h], ,
> >>> +		     AC_MSG_ERROR([Cannot find header needed for nfsdcld]))
> >>> +  fi
> >> This adds a dependency on the kernel headers being present which 
> >> is unprecedented for nfs-utils or any other package I'm aware of
> >> and I just don't think its a good idea... 
> >>
> >> So I'm wondering if there is someway around creating this
> >> dependency...
> >>
> > 
> > The idea with that header was to have a common userspace/kernel header
> > that describes the upcall/downcall format. The proper place for those
> > is generally the kernel-headers package (at least on Fedora). Other
> > distros have a similar package.
> > 
> > One possibility is to include a copy of this header with nfs-utils, and
> > only use it if /usr/include/linux/nfsd/cld.h isn't present. It's not
> > ideal, but I suppose we could live with it if adding a dependency on
> > kernel headers is anathema. It would take some rework of the code and
> > autoconf/automake stuff however.
> > 
> I believe this is how it was done in the past, keeping separate
> header files... Since I'm causing the pain, I would be more than 
> willing to do the autoconf/automake updates...
> 

Ok. If we ever have to rev the upcall format, then we'll also need to
fix the daemon to deal with legacy formats as well. Using a header
bundled with nfs-utils might actually make that easier.

I'm rather swamped at the moment, so I'll take you up on the offer to
fix up the autoconf/automake stuff :).
-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH v5 1/9] nfsdcld: add client tracking daemon stub

[Steve Dickson]

On 04/16/2012 11:42 AM, Jeff Layton wrote:
> On Mon, 16 Apr 2012 11:14:34 -0400
> 
> I'm rather swamped at the moment, so I'll take you up on the offer to
> fix up the autoconf/automake stuff :).
Of course you will!! 8-)

Thanks!

steved.

Re: [PATCH v5 0/9] nfsdcld: add a daemon to track NFSv4 client names on stable storage

[Steve Dickson]

On 02/01/2012 10:44 AM, Jeff Layton wrote:
> This patchset is the userspace portion of the knfsd client name tracking
> overhaul. See this patch series for an explanation:
> 
>     nfsd: overhaul the client name tracking code
> 
> This version scales back some of the changes in the last set. Since
> then, it's become clear that I'm not entirely certain what changes we'll
> need in order to handle this in a clustered configuration. Instead of
> trying to anticipate those changes, I've decided just to focus on the
> single-server case and leave any other changes for later once it becomes
> more clear what (if anything) is needed.
> 
> The main changes from the last set are:
> 
> - the "remove" upcall has been readded
> 
> - the "init" upcall has been removed
> 
> - the "check" and "update" upcalls no longer return a client index
>   to the kernel
> 
> - the pipe open and reopen code has been overhauled and consolidated.
>   Instead of looping indefinitely when trying to reopen the file, it will
>   abort the daemon if the open fails with anything besides ENOENT. If
>   the pipe isn't present when the daemon starts, it now uses the inotify
>   event callback to handle the opening.
> 
> Jeff Layton (9):
>   nfsdcld: add client tracking daemon stub
>   nfsdcld: add autoconf goop for sqlite
>   nfsdcld: add routines for a sqlite backend database
>   nfsdcld: add remove functionality
>   nfsdcld: add check/update functionality
>   nfsdcld: add function to remove unreclaimed client records
>   nfsdcld: make it watch for inotify events in the containing directory
>   nfsdcld: add a manpage for nfsdcld
>   nfsdcld: update the README
Committed.... My apologies for taking so long... 

steved.