nfsv41 over AF_VSOCK (nfs-ganesha)

nfsv41 over AF_VSOCK (nfs-ganesha)

[Matt Benjamin]

Hi devs (CC Bruce--here is a use case for vmci sockets transport)

One of Sage's possible plans for Manilla integration would use nfs over the new Linux  vmci sockets transport integration in qemu (below) to access Cephfs via an nfs-ganesha server running in the host vm.

This now experimentally works.

some notes on running nfs-ganesha over AF_VSOCK:

1. need stefan hajnoczi's patches for
* linux kernel (and build w/vhost-vsock support
* qemu (and build w/vhost-vsock support)
* nfs-utils (in vm guest)

all linked from https://github.com/stefanha?tab=repositories

2. host and vm guest kernels must include vhost-vsock
* host kernel should load vhost-vsock.ko

3. start a qemu(-kvm) guest (w/patched kernel) with a vhost-vsock-pci device, e.g

/opt/qemu-vsock/bin/qemu-system-x86_64    -m 2048 -usb -name vsock1     --enable-kvm     -drive file=/opt/images/vsock.qcow,if=virtio,index=0,format=qcow2     -drive file=/opt/isos/f22.iso,media=cdrom     -net nic,model=virtio,macaddr=02:36:3e:41:1b:78     -net bridge,br=br0     -parallel none     -serial mon:stdio -device vhost-vsock-pci,id=vhost-vsock-pci0,addr=4.0,guest-cid=4  -boot c

4. nfs-gansha (in host)
* need nfs-ganesha and its ntirpc rpc provider with vsock support
https://github.com/linuxbox2/nfs-ganesha (vsock branch)
https://github.com/linuxbox2/ntirpc (vsock branch)

* configure ganesha w/vsock support
cmake -DCMAKE_INSTALL_PREFIX=/cache/nfs-vsock -DUSE_FSAL_VFS=ON -DUSE_VSOCK -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" ../src

in ganesha.conf, add "nfsvsock" to Protocols list in EXPORT block

5. mount in guest w/nfs41:
(e.g., in fstab)
2:// /vsock41 nfs noauto,soft,nfsvers=4.1,sec=sys,proto=vsock,clientaddr=4,rsize=1048576,wsize=1048576 0 0

If you try this, send feedback.

Thanks!

Matt

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[Stefan Hajnoczi]

On Fri, Oct 16, 2015 at 05:08:17PM -0400, Matt Benjamin wrote:
> One of Sage's possible plans for Manilla integration would use nfs over the new Linux  vmci sockets transport integration in qemu (below) to access Cephfs via an nfs-ganesha server running in the host vm.

Excellent job!  Nice to see you were able to add AF_VSOCK support to
nfs-ganesha so quickly.

I'm currently working on kernel nfsd support and will send the patches
to linux-nfs and CC you.

Stefan

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[J. Bruce Fields]

On Fri, Oct 16, 2015 at 05:08:17PM -0400, Matt Benjamin wrote:
> Hi devs (CC Bruce--here is a use case for vmci sockets transport)
> 
> One of Sage's possible plans for Manilla integration would use nfs over the new Linux  vmci sockets transport integration in qemu (below) to access Cephfs via an nfs-ganesha server running in the host vm.

What does "the host vm" mean, and why is this a particularly useful
configuration?

--b.

> 
> This now experimentally works.
> 
> some notes on running nfs-ganesha over AF_VSOCK:
> 
> 1. need stefan hajnoczi's patches for
> * linux kernel (and build w/vhost-vsock support
> * qemu (and build w/vhost-vsock support)
> * nfs-utils (in vm guest)
> 
> all linked from https://github.com/stefanha?tab=repositories
> 
> 2. host and vm guest kernels must include vhost-vsock
> * host kernel should load vhost-vsock.ko
> 
> 3. start a qemu(-kvm) guest (w/patched kernel) with a vhost-vsock-pci device, e.g
> 
> /opt/qemu-vsock/bin/qemu-system-x86_64    -m 2048 -usb -name vsock1     --enable-kvm     -drive file=/opt/images/vsock.qcow,if=virtio,index=0,format=qcow2     -drive file=/opt/isos/f22.iso,media=cdrom     -net nic,model=virtio,macaddr=02:36:3e:41:1b:78     -net bridge,br=br0     -parallel none     -serial mon:stdio -device vhost-vsock-pci,id=vhost-vsock-pci0,addr=4.0,guest-cid=4  -boot c
> 
> 4. nfs-gansha (in host)
> * need nfs-ganesha and its ntirpc rpc provider with vsock support
> https://github.com/linuxbox2/nfs-ganesha (vsock branch)
> https://github.com/linuxbox2/ntirpc (vsock branch)
> 
> * configure ganesha w/vsock support
> cmake -DCMAKE_INSTALL_PREFIX=/cache/nfs-vsock -DUSE_FSAL_VFS=ON -DUSE_VSOCK -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" ../src
> 
> in ganesha.conf, add "nfsvsock" to Protocols list in EXPORT block
> 
> 5. mount in guest w/nfs41:
> (e.g., in fstab)
> 2:// /vsock41 nfs noauto,soft,nfsvers=4.1,sec=sys,proto=vsock,clientaddr=4,rsize=1048576,wsize=1048576 0 0
> 
> If you try this, send feedback.
> 
> Thanks!
> 
> Matt
> 
> -- 
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
> 
> http://www.redhat.com/en/technologies/storage
> 
> tel.  734-707-0660
> fax.  734-769-8938
> cel.  734-216-5309
> 

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[Matt Benjamin]

Hi Bruce,

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

----- Original Message -----
> From: "J. Bruce Fields" <bfields@redhat.com>
> To: "Matt Benjamin" <mbenjamin@redhat.com>
> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>, "Stefan Hajnoczi" <stefanha@redhat.com>, "Sage Weil"
> <sweil@redhat.com>
> Sent: Monday, October 19, 2015 11:13:52 AM
> Subject: Re: nfsv41 over AF_VSOCK (nfs-ganesha)
> 
> On Fri, Oct 16, 2015 at 05:08:17PM -0400, Matt Benjamin wrote:
> > Hi devs (CC Bruce--here is a use case for vmci sockets transport)
> > 
> > One of Sage's possible plans for Manilla integration would use nfs over the
> > new Linux  vmci sockets transport integration in qemu (below) to access
> > Cephfs via an nfs-ganesha server running in the host vm.
> 
> What does "the host vm" mean, and why is this a particularly useful
> configuration?

Sorry, I should say, "the vm host."

I think the claimed utility here is (at least) three-fold:

1. simplified configuration on host and guests
2. some claim to improved security through isolation
3. some expectation of improved latency/performance wrt TCP

Stefan sent a link to a set of slides with his original patches.  Did you get a chance to read through those?

[1] http://events.linuxfoundation.org/sites/events/files/slides/stefanha-kvm-forum-2015.pdf

Regards,

Matt

> 
> --b.
> 
> > 
> > This now experimentally works.
> > 
> > some notes on running nfs-ganesha over AF_VSOCK:
> > 
> > 1. need stefan hajnoczi's patches for
> > * linux kernel (and build w/vhost-vsock support
> > * qemu (and build w/vhost-vsock support)
> > * nfs-utils (in vm guest)
> > 
> > all linked from https://github.com/stefanha?tab=repositories
> > 
> > 2. host and vm guest kernels must include vhost-vsock
> > * host kernel should load vhost-vsock.ko
> > 
> > 3. start a qemu(-kvm) guest (w/patched kernel) with a vhost-vsock-pci
> > device, e.g
> > 
> > /opt/qemu-vsock/bin/qemu-system-x86_64    -m 2048 -usb -name vsock1
> > --enable-kvm     -drive
> > file=/opt/images/vsock.qcow,if=virtio,index=0,format=qcow2     -drive
> > file=/opt/isos/f22.iso,media=cdrom     -net
> > nic,model=virtio,macaddr=02:36:3e:41:1b:78     -net bridge,br=br0
> > -parallel none     -serial mon:stdio -device
> > vhost-vsock-pci,id=vhost-vsock-pci0,addr=4.0,guest-cid=4  -boot c
> > 
> > 4. nfs-gansha (in host)
> > * need nfs-ganesha and its ntirpc rpc provider with vsock support
> > https://github.com/linuxbox2/nfs-ganesha (vsock branch)
> > https://github.com/linuxbox2/ntirpc (vsock branch)
> > 
> > * configure ganesha w/vsock support
> > cmake -DCMAKE_INSTALL_PREFIX=/cache/nfs-vsock -DUSE_FSAL_VFS=ON -DUSE_VSOCK
> > -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" ../src
> > 
> > in ganesha.conf, add "nfsvsock" to Protocols list in EXPORT block
> > 
> > 5. mount in guest w/nfs41:
> > (e.g., in fstab)
> > 2:// /vsock41 nfs
> > noauto,soft,nfsvers=4.1,sec=sys,proto=vsock,clientaddr=4,rsize=1048576,wsize=1048576
> > 0 0
> > 
> > If you try this, send feedback.
> > 
> > Thanks!
> > 
> > Matt
> > 
> > --
> > Matt Benjamin
> > Red Hat, Inc.
> > 315 West Huron Street, Suite 140A
> > Ann Arbor, Michigan 48103
> > 
> > http://www.redhat.com/en/technologies/storage
> > 
> > tel.  734-707-0660
> > fax.  734-769-8938
> > cel.  734-216-5309
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[J. Bruce Fields]

On Mon, Oct 19, 2015 at 11:49:15AM -0400, Matt Benjamin wrote:
> ----- Original Message -----
> > From: "J. Bruce Fields" <bfields@redhat.com>
...
> > 
> > On Fri, Oct 16, 2015 at 05:08:17PM -0400, Matt Benjamin wrote:
> > > Hi devs (CC Bruce--here is a use case for vmci sockets transport)
> > > 
> > > One of Sage's possible plans for Manilla integration would use nfs over the
> > > new Linux  vmci sockets transport integration in qemu (below) to access
> > > Cephfs via an nfs-ganesha server running in the host vm.
> > 
> > What does "the host vm" mean, and why is this a particularly useful
> > configuration?
> 
> Sorry, I should say, "the vm host."

Got it, thanks!

> I think the claimed utility here is (at least) three-fold:
> 
> 1. simplified configuration on host and guests
> 2. some claim to improved security through isolation

So why is it especially interesting to put Ceph inside the VM and
Ganesha outside?

> 3. some expectation of improved latency/performance wrt TCP
> 
> Stefan sent a link to a set of slides with his original patches.  Did you get a chance to read through those?
> 
> [1] http://events.linuxfoundation.org/sites/events/files/slides/stefanha-kvm-forum-2015.pdf

Yep, thanks.--b.

> 
> Regards,
> 
> Matt
> 
> > 
> > --b.
> > 
> > > 
> > > This now experimentally works.
> > > 
> > > some notes on running nfs-ganesha over AF_VSOCK:
> > > 
> > > 1. need stefan hajnoczi's patches for
> > > * linux kernel (and build w/vhost-vsock support
> > > * qemu (and build w/vhost-vsock support)
> > > * nfs-utils (in vm guest)
> > > 
> > > all linked from https://github.com/stefanha?tab=repositories
> > > 
> > > 2. host and vm guest kernels must include vhost-vsock
> > > * host kernel should load vhost-vsock.ko
> > > 
> > > 3. start a qemu(-kvm) guest (w/patched kernel) with a vhost-vsock-pci
> > > device, e.g
> > > 
> > > /opt/qemu-vsock/bin/qemu-system-x86_64    -m 2048 -usb -name vsock1
> > > --enable-kvm     -drive
> > > file=/opt/images/vsock.qcow,if=virtio,index=0,format=qcow2     -drive
> > > file=/opt/isos/f22.iso,media=cdrom     -net
> > > nic,model=virtio,macaddr=02:36:3e:41:1b:78     -net bridge,br=br0
> > > -parallel none     -serial mon:stdio -device
> > > vhost-vsock-pci,id=vhost-vsock-pci0,addr=4.0,guest-cid=4  -boot c
> > > 
> > > 4. nfs-gansha (in host)
> > > * need nfs-ganesha and its ntirpc rpc provider with vsock support
> > > https://github.com/linuxbox2/nfs-ganesha (vsock branch)
> > > https://github.com/linuxbox2/ntirpc (vsock branch)
> > > 
> > > * configure ganesha w/vsock support
> > > cmake -DCMAKE_INSTALL_PREFIX=/cache/nfs-vsock -DUSE_FSAL_VFS=ON -DUSE_VSOCK
> > > -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" ../src
> > > 
> > > in ganesha.conf, add "nfsvsock" to Protocols list in EXPORT block
> > > 
> > > 5. mount in guest w/nfs41:
> > > (e.g., in fstab)
> > > 2:// /vsock41 nfs
> > > noauto,soft,nfsvers=4.1,sec=sys,proto=vsock,clientaddr=4,rsize=1048576,wsize=1048576
> > > 0 0
> > > 
> > > If you try this, send feedback.
> > > 
> > > Thanks!
> > > 
> > > Matt
> > > 
> > > --
> > > Matt Benjamin
> > > Red Hat, Inc.
> > > 315 West Huron Street, Suite 140A
> > > Ann Arbor, Michigan 48103
> > > 
> > > http://www.redhat.com/en/technologies/storage
> > > 
> > > tel.  734-707-0660
> > > fax.  734-769-8938
> > > cel.  734-216-5309
> > > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[Matt Benjamin]

Hi Bruce,

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

----- Original Message -----
> From: "J. Bruce Fields" <bfields@redhat.com>
> To: "Matt Benjamin" <mbenjamin@redhat.com>
> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>, "Stefan Hajnoczi" <stefanha@redhat.com>, "Sage Weil"
> <sweil@redhat.com>
> Sent: Monday, October 19, 2015 11:58:45 AM
> Subject: Re: nfsv41 over AF_VSOCK (nfs-ganesha)
> 
> On Mon, Oct 19, 2015 at 11:49:15AM -0400, Matt Benjamin wrote:
> > ----- Original Message -----
> > > From: "J. Bruce Fields" <bfields@redhat.com>
> ...
> > > 
> > > On Fri, Oct 16, 2015 at 05:08:17PM -0400, Matt Benjamin wrote:
> > > > Hi devs (CC Bruce--here is a use case for vmci sockets transport)
> > > > 
> > > > One of Sage's possible plans for Manilla integration would use nfs over
> > > > the
> > > > new Linux  vmci sockets transport integration in qemu (below) to access
> > > > Cephfs via an nfs-ganesha server running in the host vm.
> > > 
> > > What does "the host vm" mean, and why is this a particularly useful
> > > configuration?
> > 
> > Sorry, I should say, "the vm host."
> 
> Got it, thanks!
> 
> > I think the claimed utility here is (at least) three-fold:
> > 
> > 1. simplified configuration on host and guests
> > 2. some claim to improved security through isolation
> 
> So why is it especially interesting to put Ceph inside the VM and
> Ganesha outside?

Oh, sorry.  Here Ceph (or Gluster, or, whatever underlying FS provider) is conceptually outside the vm complex altogether, Ganesha is re-exporting on the vm host, and guests access the namespace using NFS(v41).

Regards,

Matt

> 
> > 3. some expectation of improved latency/performance wrt TCP
> > 
> > Stefan sent a link to a set of slides with his original patches.  Did you
> > get a chance to read through those?
> > 
> > [1]
> > http://events.linuxfoundation.org/sites/events/files/slides/stefanha-kvm-forum-2015.pdf
> 
> Yep, thanks.--b.
> 
> > 
> > Regards,
> > 
> > Matt
> > 
> > > 
> > > --b.
> > > 
> > > > 
> > > > This now experimentally works.
> > > > 
> > > > some notes on running nfs-ganesha over AF_VSOCK:
> > > > 
> > > > 1. need stefan hajnoczi's patches for
> > > > * linux kernel (and build w/vhost-vsock support
> > > > * qemu (and build w/vhost-vsock support)
> > > > * nfs-utils (in vm guest)
> > > > 
> > > > all linked from https://github.com/stefanha?tab=repositories
> > > > 
> > > > 2. host and vm guest kernels must include vhost-vsock
> > > > * host kernel should load vhost-vsock.ko
> > > > 
> > > > 3. start a qemu(-kvm) guest (w/patched kernel) with a vhost-vsock-pci
> > > > device, e.g
> > > > 
> > > > /opt/qemu-vsock/bin/qemu-system-x86_64    -m 2048 -usb -name vsock1
> > > > --enable-kvm     -drive
> > > > file=/opt/images/vsock.qcow,if=virtio,index=0,format=qcow2     -drive
> > > > file=/opt/isos/f22.iso,media=cdrom     -net
> > > > nic,model=virtio,macaddr=02:36:3e:41:1b:78     -net bridge,br=br0
> > > > -parallel none     -serial mon:stdio -device
> > > > vhost-vsock-pci,id=vhost-vsock-pci0,addr=4.0,guest-cid=4  -boot c
> > > > 
> > > > 4. nfs-gansha (in host)
> > > > * need nfs-ganesha and its ntirpc rpc provider with vsock support
> > > > https://github.com/linuxbox2/nfs-ganesha (vsock branch)
> > > > https://github.com/linuxbox2/ntirpc (vsock branch)
> > > > 
> > > > * configure ganesha w/vsock support
> > > > cmake -DCMAKE_INSTALL_PREFIX=/cache/nfs-vsock -DUSE_FSAL_VFS=ON
> > > > -DUSE_VSOCK
> > > > -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" ../src
> > > > 
> > > > in ganesha.conf, add "nfsvsock" to Protocols list in EXPORT block
> > > > 
> > > > 5. mount in guest w/nfs41:
> > > > (e.g., in fstab)
> > > > 2:// /vsock41 nfs
> > > > noauto,soft,nfsvers=4.1,sec=sys,proto=vsock,clientaddr=4,rsize=1048576,wsize=1048576
> > > > 0 0
> > > > 
> > > > If you try this, send feedback.
> > > > 
> > > > Thanks!
> > > > 
> > > > Matt
> > > > 
> > > > --
> > > > Matt Benjamin
> > > > Red Hat, Inc.
> > > > 315 West Huron Street, Suite 140A
> > > > Ann Arbor, Michigan 48103
> > > > 
> > > > http://www.redhat.com/en/technologies/storage
> > > > 
> > > > tel.  734-707-0660
> > > > fax.  734-769-8938
> > > > cel.  734-216-5309
> > > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[John Spray]

On Fri, Oct 16, 2015 at 10:08 PM, Matt Benjamin <mbenjamin@redhat.com> wrote:
> Hi devs (CC Bruce--here is a use case for vmci sockets transport)
>
> One of Sage's possible plans for Manilla integration would use nfs over the new Linux  vmci sockets transport integration in qemu (below) to access Cephfs via an nfs-ganesha server running in the host vm.
>
> This now experimentally works.

Very cool!  Thank you for the detailed instructions, I look forward to
trying this out soon.

John

> some notes on running nfs-ganesha over AF_VSOCK:
>
> 1. need stefan hajnoczi's patches for
> * linux kernel (and build w/vhost-vsock support
> * qemu (and build w/vhost-vsock support)
> * nfs-utils (in vm guest)
>
> all linked from https://github.com/stefanha?tab=repositories
>
> 2. host and vm guest kernels must include vhost-vsock
> * host kernel should load vhost-vsock.ko
>
> 3. start a qemu(-kvm) guest (w/patched kernel) with a vhost-vsock-pci device, e.g
>
> /opt/qemu-vsock/bin/qemu-system-x86_64    -m 2048 -usb -name vsock1     --enable-kvm     -drive file=/opt/images/vsock.qcow,if=virtio,index=0,format=qcow2     -drive file=/opt/isos/f22.iso,media=cdrom     -net nic,model=virtio,macaddr=02:36:3e:41:1b:78     -net bridge,br=br0     -parallel none     -serial mon:stdio -device vhost-vsock-pci,id=vhost-vsock-pci0,addr=4.0,guest-cid=4  -boot c
>
> 4. nfs-gansha (in host)
> * need nfs-ganesha and its ntirpc rpc provider with vsock support
> https://github.com/linuxbox2/nfs-ganesha (vsock branch)
> https://github.com/linuxbox2/ntirpc (vsock branch)
>
> * configure ganesha w/vsock support
> cmake -DCMAKE_INSTALL_PREFIX=/cache/nfs-vsock -DUSE_FSAL_VFS=ON -DUSE_VSOCK -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" ../src
>
> in ganesha.conf, add "nfsvsock" to Protocols list in EXPORT block
>
> 5. mount in guest w/nfs41:
> (e.g., in fstab)
> 2:// /vsock41 nfs noauto,soft,nfsvers=4.1,sec=sys,proto=vsock,clientaddr=4,rsize=1048576,wsize=1048576 0 0
>
> If you try this, send feedback.
>
> Thanks!
>
> Matt
>
> --
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-707-0660
> fax.  734-769-8938
> cel.  734-216-5309
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[John Spray]

On Mon, Oct 19, 2015 at 5:13 PM, John Spray <jspray@redhat.com> wrote:
>> If you try this, send feedback.
>>

OK, got this up and running.

I've shared the kernel/qemu/nfsutils packages I built here:
https://copr.fedoraproject.org/coprs/jspray/vsock-nfs/builds/

(at time of writing the kernel one is still building, and I'm running
with ganesha out of a source tree)

Observations:
 * Running VM as qemu user gives EPERM opening vsock device, even
after changing permissions on the device node (for which I guess we'll
want udev rules at some stage) -- is there a particular capability
that we need to grant the qemu user?  Was looking into this to make it
convenient to run inside libvirt.
 * NFS writes from the guest are lagging for like a minute before
completing, my hunch is that this is something in the NFS client
recovery stuff (in ganesha) that's not coping with vsock, the
operations seem to complete at the point where the server declares
itself "NOT IN GRACE".
 * For those (like myself) unaccustomed to running ganesha, do not run
it straight out of a source tree and expect everything to work, by
default even VFS exports won't work that way (mounts work but clients
see an empty tree) because it can't find the built FSAL .so.  You can
write a config file that works, but it's easier just to make install
it.
 * (Anecdotal, seen while messing with other stuff) client mount seems
to hang if I kill ganesha and then start it again, not sure if this is
a ganesha issue or a general vsock issue.

Cheers,
John

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[Daniel Gryniewicz]

On Fri, Oct 23, 2015 at 9:27 AM, John Spray <jspray@redhat.com> wrote:
>  * NFS writes from the guest are lagging for like a minute before
> completing, my hunch is that this is something in the NFS client
> recovery stuff (in ganesha) that's not coping with vsock, the
> operations seem to complete at the point where the server declares
> itself "NOT IN GRACE".


Ganesha always starts in Grace, and will not process new clients until
it exits Grace.  Existing clients should re-connect fine, and new
clients work fine after Grace is exited.

Dan

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[Matt Benjamin]

For hacking around, put  "Graceless = true;" in the NFSV4 block.

Matt

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

----- Original Message -----
> From: "Daniel Gryniewicz" <dang@redhat.com>
> To: "John Spray" <jspray@redhat.com>
> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>, "Stefan Hajnoczi" <shajnocz@redhat.com>
> Sent: Friday, October 23, 2015 12:34:42 PM
> Subject: Re: nfsv41 over AF_VSOCK (nfs-ganesha)
> 
> On Fri, Oct 23, 2015 at 9:27 AM, John Spray <jspray@redhat.com> wrote:
> >  * NFS writes from the guest are lagging for like a minute before
> > completing, my hunch is that this is something in the NFS client
> > recovery stuff (in ganesha) that's not coping with vsock, the
> > operations seem to complete at the point where the server declares
> > itself "NOT IN GRACE".
> 
> 
> Ganesha always starts in Grace, and will not process new clients until
> it exits Grace.  Existing clients should re-connect fine, and new
> clients work fine after Grace is exited.
> 
> Dan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: nfsv41 over AF_VSOCK (nfs-ganesha)

[Stefan Hajnoczi]

On Fri, Oct 23, 2015 at 02:27:22PM +0100, John Spray wrote:
> On Mon, Oct 19, 2015 at 5:13 PM, John Spray <jspray@redhat.com> wrote:
> >> If you try this, send feedback.
> >>
> 
> OK, got this up and running.
> 
> I've shared the kernel/qemu/nfsutils packages I built here:
> https://copr.fedoraproject.org/coprs/jspray/vsock-nfs/builds/
> 
> (at time of writing the kernel one is still building, and I'm running
> with ganesha out of a source tree)
> 
> Observations:
>  * Running VM as qemu user gives EPERM opening vsock device, even
> after changing permissions on the device node (for which I guess we'll
> want udev rules at some stage) -- is there a particular capability
> that we need to grant the qemu user?  Was looking into this to make it
> convenient to run inside libvirt.

libvirtd runs as root and opens /dev/vhost-*.  It passes file
descriptors to the unprivileged QEMU process.  I think this is how
things work in production (with SELinux enabled too).

On a development machine it is easier to either run QEMU as root or set
uid:gid on /dev/vhost-vsock.

So we don't need to do anything, although libvirt code will need to be
written to support vhost-vsock.  It should be very similar to the
existing vhost-net code in libvirt.

>  * NFS writes from the guest are lagging for like a minute before
> completing, my hunch is that this is something in the NFS client
> recovery stuff (in ganesha) that's not coping with vsock, the
> operations seem to complete at the point where the server declares
> itself "NOT IN GRACE".
>  * For those (like myself) unaccustomed to running ganesha, do not run
> it straight out of a source tree and expect everything to work, by
> default even VFS exports won't work that way (mounts work but clients
> see an empty tree) because it can't find the built FSAL .so.  You can
> write a config file that works, but it's easier just to make install
> it.
>  * (Anecdotal, seen while messing with other stuff) client mount seems
> to hang if I kill ganesha and then start it again, not sure if this is
> a ganesha issue or a general vsock issue.

If you experience hangs when the other side closes the connection you
may need:
https://github.com/stefanha/linux/commit/ae3c6c9b1534c1df5213a72f38e377ecd0852e14

Stefan