Fixing NFS over OpenVPN

Fixing NFS over OpenVPN

[Toby Goodwin]

NFS over OpenVPN is unreliable. On systems with modest RAM, it quite
quickly locks up in a nasty way, with unkillable processes, and the
system unable even to shutdown or reboot. Ouch.

I've been looking at this for some time, and have finally made some
progress. The problem is that if the openvpn process needs any more
kernel memory, the kernel will sometimes ask nfs to release some pages,
nfs calls commit_inode(), which wants to write to the openvpn tunnel,
and we have deadlock.

I believe I know how to fix this, but would appreciate some guidance.
Part of the solution is to specify the "--mlock" flag to openvpn -- this
exists so that secrets are never written to swap, but as a side effect
it prevents openvpn from ever page faulting.

The other kernel allocation culprits are the sockets that openvpn holds.
To prevent them triggering the deadlock, we need to set sk_allocation =
GFP_NOIO. For the socket created by the tun driver it's a simple change
to drivers/net/tun.c. (This is the socket that applications use when
they want to communicate with the tunnel.)

The other socket created by openvpn is trickier, as it's just a normal
userland AF_INET socket. (This is the socket through which OpenVPN sends
its encrypted packets to the remote end of the tunnel.) I haven't
discovered any way for userland to request a particular allocation
policy for a socket, please let me know if I've missed something.

So I've added a setsockopt() call that can do it:

--- net/core/sock.c.orig	2012-03-18 23:15:34.000000000 +0000
+++ net/core/sock.c	2012-04-22 22:55:34.320023377 +0100
@@ -793,6 +793,10 @@
 		sock_valbool_flag(sk, SOCK_WIFI_STATUS, valbool);
 		break;
 
+	case SO_RECURSIVE:
+		sk->sk_allocation = GFP_NOIO;
+		break;
+
 	default:
 		ret = -ENOPROTOOPT;
 		break;

(I picked the SO_RECURSIVE name late at night -- suggestions for a
better name are very welcome!)

I've dealt with these three troublesome memory allocations as I've
discovered them, and initial testing shows that I can now keep nfs over
openvpn going, even on a system with limited memory that is also
thrashing with a large compile.

But this seems like a fragile process: I don't know if there are other
troublespots lurking. Really I'd like openvpn to say to the kernel
"whenever you allocate memory for me, use GFP_NOIO", but I haven't
discovered a way to do this: none of the process flags quite seem to fit
the bill. (I did consider PF_KSWAPD, but thought better of it. :-)

Is there a better way to solve this?

Thanks,

Toby.

Re: Fixing NFS over OpenVPN

[Andi Kleen]

Toby Goodwin <toby@mythic-beasts.com> writes:
>
> I believe I know how to fix this, but would appreciate some guidance.
> Part of the solution is to specify the "--mlock" flag to openvpn -- this
> exists so that secrets are never written to swap, but as a side effect
> it prevents openvpn from ever page faulting.

mlock does not prevent page faulting. It merely prevents existing pages
getting swapped out, but new pages can be definitely be faulted in.
Your explanation would only make sense if all swappable memory 
in the system is tied up in the openvpn process, which is unlikely.

That said there may be still deadlocks if openvpn needs any new
memory for writing data, but that has nothing to do with mlock.

In fact the mlock should make it more stable under swapping, but you
still have the problem you describe next.

> userland AF_INET socket. (This is the socket through which OpenVPN sends
> its encrypted packets to the remote end of the tunnel.) I haven't
> discovered any way for userland to request a particular allocation
> policy for a socket, please let me know if I've missed something.

Just changing to GFP_NOIO will not fully solve the problem. 

You really would need a preallocated mempool of pages for the socket
(similar to Mel Gorman/ Peter Ziljstra's swap-over-NFS
patchkit). Essentially the whole IO path needs to be able to work
without allocating new memory.
>
> Is there a better way to solve this?

Probably parts of openvpn would need to move into the kernel.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only