Re: pnfs setup on Linux 4.15

From: Kanika <jr.kanika@gmail.com>
To: Benjamin Coddington <bcodding@redhat.com>
Cc: linux-nfs@vger.kernel.org
Subject: Re: pnfs setup on Linux 4.15
Date: Fri, 8 Mar 2019 12:35:48 +0530	[thread overview]
Message-ID: <CAP8UQm+tBWfiG4yVe=YihyOswGy00ZbXLMTcdQs-YG_M7FFRMg@mail.gmail.com> (raw)
In-Reply-To: <C235C237-B75C-4EED-B2F1-1F5E084B21CD@redhat.com>

On Thu, 7 Mar 2019 at 16:42, Benjamin Coddington <bcodding@redhat.com> wrote:
>
> On 7 Mar 2019, at 2:51, Kanika wrote:
>
> > Hi Benjamin,
> > Thanks for your reply. I suspected so but wanted to confirm. I tried
> > block/scsi layout as well but that isn't working either.
> >
> > Based on an old thread on this list on "how to setup pnfs for block
> > layout", I set up an iscsi target on the DS and the initiators on the MDS
> > and client. All machines are running 4.15 kernel. Client as an iscsi
> > initiator but does not mount the device. MDS also an initiator with an xfs
> > filesystem mounted on the device. This mount point is exported as an nfs
> > share from the MDS. The client mounts the exported MDS share.
> > By default LAYOUT_SCSI is used but the GETDEVICEINFO call keeps failing
> > with NFS4ERR_INVAL. As a result all reads/writes from the client are routed
> > to the MDS instead of the DS. Anything wrong with my setup?
>
> The SCSI layout needs to work with whole SCSI devices, not partitions.
>
> Try putting your XFS on /dev/sda instead of /dev/sda1.
>
> Ben
Hi Ben,
Thanks for the pointer. I tried this but still no luck, see the same
error for GETDEVICEINFO. Anything else I am missing?

On MDS:
root@ubuntu1604:~# lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
NAME   FSTYPE  SIZE MOUNTPOINT LABEL
sda        xfs           1G    /sudosrv               <------ iscsi block device
sr0           1024M
vda             64G
├─vda1 ext4     63G /
├─vda2           1K
└─vda5 swap    975M [SWAP]

#cat /etc/exports
/sudosrv        *(rw,sync,fsid=0,no_subtree_check,no_root_squash,pnfs)

On client:
root@ubuntu18_04_2:~# cat /proc/partitions
major minor  #blocks  name

  11        0    1048575 sr0
 252        0   67108864 vda
 252        1   66107392 vda1
 252        2          1 vda2
 252        5     998400 vda5
   8        0    1048576 sda

root@ubuntu18_04_2:~# cat /proc/self/mountstats
device 192.168.122.92:/ mounted on /mnt with fstype nfs4 statvers=1.1
...
nfsv4:  bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x803,acl=0x3,sessions,pnfs=LAYOUT_SCSI

blkmapd and blocklayoutdriver loaded on the client
root@ubuntu18_04_2:~# ps aux | grep blk
root      2485  0.0  0.0  23920   176 ?        Ss   21:25   0:00
/usr/sbin/blkmapd

-- packet exchange between MDS and client.
   94 2019-03-07 21:31:18.480733 192.168.122.83 → 192.168.122.92 NFS
234 V4 Call GETDEVINFO
   95 2019-03-07 21:31:18.483495 192.168.122.92 → 192.168.122.83 NFS
158 V4 Reply (Call In 94) GETDEVINFO Status: NFS4ERR_INVAL
   96 2019-03-07 21:31:18.483878 192.168.122.83 → 192.168.122.92 NFS
282 V4 Call LAYOUTRETURN
   98 2019-03-07 21:31:18.483935 192.168.122.83 → 192.168.122.92 NFS
1478 V4 Call WRITE StateID: 0xcaae Offset: 0 Len: 4096
  100 2019-03-07 21:31:18.484204 192.168.122.92 → 192.168.122.83 NFS
170 V4 Reply (Call In 96) LAYOUTRETURN
  102 2019-03-07 21:31:18.528822 192.168.122.92 → 192.168.122.83 NFS
246 V4 Reply (Call In 98) WRITE