* Proposal for simplifying NFS/RDMA client memory registration @ 2014-02-26 16:44 ` Chuck Lever 0 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-02-26 16:44 UTC (permalink / raw) To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List; +Cc: Shirley Ma Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. Not all HCAs support all memory registration modes, so our test matrix is quite large. I’d like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can’t test. BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I’m told it was never intended for production use. REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. I propose removing BOUNCEBUFFERS since it is not intended for production use. I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. I propose removing MTHCA_FMR as I’m told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. Thoughts/comments? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Proposal for simplifying NFS/RDMA client memory registration @ 2014-02-26 16:44 ` Chuck Lever 0 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-02-26 16:44 UTC (permalink / raw) To: linux-rdma, Linux NFS Mailing List; +Cc: Shirley Ma Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. Not all HCAs support all memory registration modes, so our test matrix is quite large. I’d like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can’t test. BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I’m told it was never intended for production use. REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. I propose removing BOUNCEBUFFERS since it is not intended for production use. I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. I propose removing MTHCA_FMR as I’m told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. Thoughts/comments? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <01C4496A-F074-4F72-9DF0-6076C05E8A1F-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>]
* RE: Proposal for simplifying NFS/RDMA client memory registration 2014-02-26 16:44 ` Chuck Lever @ 2014-02-26 17:42 ` faibish, sorin -1 siblings, 0 replies; 26+ messages in thread From: faibish, sorin @ 2014-02-26 17:42 UTC (permalink / raw) To: Chuck Lever, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List Cc: Shirley Ma -----Original Message----- From: linux-nfs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-nfs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Chuck Lever Sent: Wednesday, February 26, 2014 11:45 AM To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Linux NFS Mailing List Cc: Shirley Ma Subject: Proposal for simplifying NFS/RDMA client memory registration Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we've built and run functional tests to determine what is working and what is broken. One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. Not all HCAs support all memory registration modes, so our test matrix is quite large. I'd like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can't test. BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I'm told it was never intended for production use. REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. I propose removing BOUNCEBUFFERS since it is not intended for production use. I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. I propose removing MTHCA_FMR as I'm told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. [sf] We need volunteers for server too as the optimizations on client only may not be enough considering that SMB express optimized both sides. Maybe we can find a student of Erez to work on this. Thanks ./Sorin Thoughts/comments? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Proposal for simplifying NFS/RDMA client memory registration @ 2014-02-26 17:42 ` faibish, sorin 0 siblings, 0 replies; 26+ messages in thread From: faibish, sorin @ 2014-02-26 17:42 UTC (permalink / raw) To: Chuck Lever, linux-rdma, Linux NFS Mailing List; +Cc: Shirley Ma -----Original Message----- From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-owner@vger.kernel.org] On Behalf Of Chuck Lever Sent: Wednesday, February 26, 2014 11:45 AM To: linux-rdma@vger.kernel.org; Linux NFS Mailing List Cc: Shirley Ma Subject: Proposal for simplifying NFS/RDMA client memory registration Hi- Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we've built and run functional tests to determine what is working and what is broken. One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. Not all HCAs support all memory registration modes, so our test matrix is quite large. I'd like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can't test. BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I'm told it was never intended for production use. REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. I propose removing BOUNCEBUFFERS since it is not intended for production use. I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. I propose removing MTHCA_FMR as I'm told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. [sf] We need volunteers for server too as the optimizations on client only may not be enough considering that SMB express optimized both sides. Maybe we can find a student of Erez to work on this. Thanks ./Sorin Thoughts/comments? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-02-26 16:44 ` Chuck Lever @ 2014-02-28 21:41 ` Tom Talpey -1 siblings, 0 replies; 26+ messages in thread From: Tom Talpey @ 2014-02-28 21:41 UTC (permalink / raw) To: Chuck Lever, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List Cc: Shirley Ma On 2/26/2014 8:44 AM, Chuck Lever wrote: > Hi- > > Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. > > One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. > > Not all HCAs support all memory registration modes, so our test matrix is quite large. I’d like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can’t test. > > BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I’m told it was never intended for production use. > > REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. > > MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. > > MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. > > MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. The MTHCA FMR is not completely safe - it protects only on page boundaries, therefore the neighboring bytes are vulnerable to silent corruption (reads) and exposure (write). It is quite correct that they are supported on only a specific set of legacy Mellanox HCA. You should consider removing the code that looked for this PCI ID and attempted to alter the device's wire MTU, to overcome another of its limitations. > > FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. This should be, by far, the preferred mode. Also, if I recall correctly, the server depends on this mode being available/supported. However, it may not be supported by Soft iWARP. Physical addressing is used. > > ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. Not safe is an understatement. It exposes all of client physical memory to the peer, for both read and write. A simple pointer error on the server will silently corrupt the client. This mode was intended only for testing, and in experimental deployments. Tom. > > > I propose removing BOUNCEBUFFERS since it is not intended for production use. > > I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. > > I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. > > I propose removing MTHCA_FMR as I’m told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. > > This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. > > We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. > > Thoughts/comments? > > -- > Chuck Lever > chuck[dot]lever[at]oracle[dot]com > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-02-28 21:41 ` Tom Talpey 0 siblings, 0 replies; 26+ messages in thread From: Tom Talpey @ 2014-02-28 21:41 UTC (permalink / raw) To: Chuck Lever, linux-rdma, Linux NFS Mailing List; +Cc: Shirley Ma On 2/26/2014 8:44 AM, Chuck Lever wrote: > Hi- > > Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. > > One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. > > Not all HCAs support all memory registration modes, so our test matrix is quite large. I’d like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can’t test. > > BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I’m told it was never intended for production use. > > REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. > > MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. > > MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. > > MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. The MTHCA FMR is not completely safe - it protects only on page boundaries, therefore the neighboring bytes are vulnerable to silent corruption (reads) and exposure (write). It is quite correct that they are supported on only a specific set of legacy Mellanox HCA. You should consider removing the code that looked for this PCI ID and attempted to alter the device's wire MTU, to overcome another of its limitations. > > FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. This should be, by far, the preferred mode. Also, if I recall correctly, the server depends on this mode being available/supported. However, it may not be supported by Soft iWARP. Physical addressing is used. > > ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. Not safe is an understatement. It exposes all of client physical memory to the peer, for both read and write. A simple pointer error on the server will silently corrupt the client. This mode was intended only for testing, and in experimental deployments. Tom. > > > I propose removing BOUNCEBUFFERS since it is not intended for production use. > > I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. > > I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. > > I propose removing MTHCA_FMR as I’m told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. > > This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. > > We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. > > Thoughts/comments? > > -- > Chuck Lever > chuck[dot]lever[at]oracle[dot]com > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <CABgxfbHwo13ma=-Xn+S36WwD8LVNLdw6UHztFe8EkCA_=NBenw@mail.gmail.com>]
[parent not found: <CABgxfbHwo13ma=-Xn+S36WwD8LVNLdw6UHztFe8EkCA_=NBenw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration [not found] ` <CABgxfbHwo13ma=-Xn+S36WwD8LVNLdw6UHztFe8EkCA_=NBenw@mail.gmail.com> @ 2014-02-28 22:26 ` Wendy Cheng 0 siblings, 0 replies; 26+ messages in thread From: Wendy Cheng @ 2014-02-28 22:26 UTC (permalink / raw) To: Tom Talpey Cc: Chuck Lever, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote: >> >> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>> >>> Hi- >>> >>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>> the Linux kernel. So far we've built and run functional tests to determine >>> what is working and what is broken. >>> >>> [snip] > > >>> >>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>> All HCAs support this mode. >> >> >> Not safe is an understatement. It exposes all of client physical >> memory to the peer, for both read and write. A simple pointer error >> on the server will silently corrupt the client. This mode was >> intended only for testing, and in experimental deployments. (sorry, resend .. previous reply bounced back due to gmail html format) Please keep "ALLPHYSICAL" for now - as our embedded system needs it. Thanks, Wendy -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-02-28 22:26 ` Wendy Cheng 0 siblings, 0 replies; 26+ messages in thread From: Wendy Cheng @ 2014-02-28 22:26 UTC (permalink / raw) To: Tom Talpey; +Cc: Chuck Lever, linux-rdma, Linux NFS Mailing List, Shirley Ma On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: > ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom@talpey.com> wrote: >> >> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>> >>> Hi- >>> >>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>> the Linux kernel. So far we've built and run functional tests to determine >>> what is working and what is broken. >>> >>> [snip] > > >>> >>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>> All HCAs support this mode. >> >> >> Not safe is an understatement. It exposes all of client physical >> memory to the peer, for both read and write. A simple pointer error >> on the server will silently corrupt the client. This mode was >> intended only for testing, and in experimental deployments. (sorry, resend .. previous reply bounced back due to gmail html format) Please keep "ALLPHYSICAL" for now - as our embedded system needs it. Thanks, Wendy ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <CABgxfbHg0B02c2zpYdCB4pUZZhhA4aKRuyh4Kx=NE1yAisFLLQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-02-28 22:26 ` Wendy Cheng @ 2014-03-01 2:59 ` Chuck Lever -1 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-01 2:59 UTC (permalink / raw) To: Wendy Cheng Cc: Tom Talpey, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma Hi Wendy- On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote: >>> >>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>> >>>> Hi- >>>> >>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>> the Linux kernel. So far we've built and run functional tests to determine >>>> what is working and what is broken. >>>> >>>> [snip] >> >> >>>> >>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>> All HCAs support this mode. >>> >>> >>> Not safe is an understatement. It exposes all of client physical >>> memory to the peer, for both read and write. A simple pointer error >>> on the server will silently corrupt the client. This mode was >>> intended only for testing, and in experimental deployments. > > (sorry, resend .. previous reply bounced back due to gmail html format) > > Please keep "ALLPHYSICAL" for now - as our embedded system needs it. This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-01 2:59 ` Chuck Lever 0 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-01 2:59 UTC (permalink / raw) To: Wendy Cheng; +Cc: Tom Talpey, linux-rdma, Linux NFS Mailing List, Shirley Ma Hi Wendy- On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: > On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom@talpey.com> wrote: >>> >>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>> >>>> Hi- >>>> >>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>> the Linux kernel. So far we've built and run functional tests to determine >>>> what is working and what is broken. >>>> >>>> [snip] >> >> >>>> >>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>> All HCAs support this mode. >>> >>> >>> Not safe is an understatement. It exposes all of client physical >>> memory to the peer, for both read and write. A simple pointer error >>> on the server will silently corrupt the client. This mode was >>> intended only for testing, and in experimental deployments. > > (sorry, resend .. previous reply bounced back due to gmail html format) > > Please keep "ALLPHYSICAL" for now - as our embedded system needs it. This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <B83B4DA6-E9CF-4DF3-9227-5EF9B4D25F4D-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-03-01 2:59 ` Chuck Lever @ 2014-03-01 16:00 ` Jeff Layton -1 siblings, 0 replies; 26+ messages in thread From: Jeff Layton @ 2014-03-01 16:00 UTC (permalink / raw) To: Chuck Lever Cc: Wendy Cheng, Tom Talpey, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma On Fri, 28 Feb 2014 21:59:00 -0500 Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote: > Hi Wendy- > > On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > > > On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: > >> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote: > >>> > >>> On 2/26/2014 8:44 AM, Chuck Lever wrote: > >>>> > >>>> Hi- > >>>> > >>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in > >>>> the Linux kernel. So far we've built and run functional tests to determine > >>>> what is working and what is broken. > >>>> > >>>> [snip] > >> > >> > >>>> > >>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. > >>>> All HCAs support this mode. > >>> > >>> > >>> Not safe is an understatement. It exposes all of client physical > >>> memory to the peer, for both read and write. A simple pointer error > >>> on the server will silently corrupt the client. This mode was > >>> intended only for testing, and in experimental deployments. > > > > (sorry, resend .. previous reply bounced back due to gmail html format) > > > > Please keep "ALLPHYSICAL" for now - as our embedded system needs it. > > This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. > > Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? > Also, given that ALLPHYSICAL isn't considered safe, we should at the very least require some sort of explicit opt-in before allowing it to be used. Perhaps either a Kconfig option, or maybe a runtime switch like a module parm? Wendy, would that be acceptable? -- Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-01 16:00 ` Jeff Layton 0 siblings, 0 replies; 26+ messages in thread From: Jeff Layton @ 2014-03-01 16:00 UTC (permalink / raw) To: Chuck Lever Cc: Wendy Cheng, Tom Talpey, linux-rdma, Linux NFS Mailing List, Shirley Ma On Fri, 28 Feb 2014 21:59:00 -0500 Chuck Lever <chuck.lever@oracle.com> wrote: > Hi Wendy- > > On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: > > > On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: > >> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom@talpey.com> wrote: > >>> > >>> On 2/26/2014 8:44 AM, Chuck Lever wrote: > >>>> > >>>> Hi- > >>>> > >>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in > >>>> the Linux kernel. So far we've built and run functional tests to determine > >>>> what is working and what is broken. > >>>> > >>>> [snip] > >> > >> > >>>> > >>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. > >>>> All HCAs support this mode. > >>> > >>> > >>> Not safe is an understatement. It exposes all of client physical > >>> memory to the peer, for both read and write. A simple pointer error > >>> on the server will silently corrupt the client. This mode was > >>> intended only for testing, and in experimental deployments. > > > > (sorry, resend .. previous reply bounced back due to gmail html format) > > > > Please keep "ALLPHYSICAL" for now - as our embedded system needs it. > > This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. > > Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? > Also, given that ALLPHYSICAL isn't considered safe, we should at the very least require some sort of explicit opt-in before allowing it to be used. Perhaps either a Kconfig option, or maybe a runtime switch like a module parm? Wendy, would that be acceptable? -- Jeff Layton <jlayton@redhat.com> ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <20140301110022.417eb088-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-03-01 16:00 ` Jeff Layton @ 2014-03-01 17:14 ` Chuck Lever -1 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-01 17:14 UTC (permalink / raw) To: Jeff Layton Cc: Wendy Cheng, Tom Talpey, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma On Mar 1, 2014, at 11:00 AM, Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote: > On Fri, 28 Feb 2014 21:59:00 -0500 > Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote: > >> Hi Wendy- >> >> On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> >>> On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >>>> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote: >>>>> >>>>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>>>> >>>>>> Hi- >>>>>> >>>>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>>>> the Linux kernel. So far we've built and run functional tests to determine >>>>>> what is working and what is broken. >>>>>> >>>>>> [snip] >>>> >>>> >>>>>> >>>>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>>>> All HCAs support this mode. >>>>> >>>>> >>>>> Not safe is an understatement. It exposes all of client physical >>>>> memory to the peer, for both read and write. A simple pointer error >>>>> on the server will silently corrupt the client. This mode was >>>>> intended only for testing, and in experimental deployments. >>> >>> (sorry, resend .. previous reply bounced back due to gmail html format) >>> >>> Please keep "ALLPHYSICAL" for now - as our embedded system needs it. >> >> This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. >> >> Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? >> > > Also, given that ALLPHYSICAL isn't considered safe, we should at the > very least require some sort of explicit opt-in before allowing it to be > used. Perhaps either a Kconfig option, or maybe a runtime switch like a > module parm? Well, there is already an opt-in: /proc/sys/sunrpc/rdma_memreg_strategy selects the default registration mode. Currently FRMR is always used unless the HCA doesn’t support it. If we need to keep ALLPHYSICAL, the least thing I would want to do is remove the logic in rpcrdma_ia_open() that switches to ALLPHYSICAL if the mode selected by rdma_memreg_strategy isn’t supported by the HCA. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-01 17:14 ` Chuck Lever 0 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-01 17:14 UTC (permalink / raw) To: Jeff Layton Cc: Wendy Cheng, Tom Talpey, linux-rdma, Linux NFS Mailing List, Shirley Ma On Mar 1, 2014, at 11:00 AM, Jeff Layton <jlayton@redhat.com> wrote: > On Fri, 28 Feb 2014 21:59:00 -0500 > Chuck Lever <chuck.lever@oracle.com> wrote: > >> Hi Wendy- >> >> On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >> >>> On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >>>> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom@talpey.com> wrote: >>>>> >>>>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>>>> >>>>>> Hi- >>>>>> >>>>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>>>> the Linux kernel. So far we've built and run functional tests to determine >>>>>> what is working and what is broken. >>>>>> >>>>>> [snip] >>>> >>>> >>>>>> >>>>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>>>> All HCAs support this mode. >>>>> >>>>> >>>>> Not safe is an understatement. It exposes all of client physical >>>>> memory to the peer, for both read and write. A simple pointer error >>>>> on the server will silently corrupt the client. This mode was >>>>> intended only for testing, and in experimental deployments. >>> >>> (sorry, resend .. previous reply bounced back due to gmail html format) >>> >>> Please keep "ALLPHYSICAL" for now - as our embedded system needs it. >> >> This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. >> >> Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? >> > > Also, given that ALLPHYSICAL isn't considered safe, we should at the > very least require some sort of explicit opt-in before allowing it to be > used. Perhaps either a Kconfig option, or maybe a runtime switch like a > module parm? Well, there is already an opt-in: /proc/sys/sunrpc/rdma_memreg_strategy selects the default registration mode. Currently FRMR is always used unless the HCA doesn’t support it. If we need to keep ALLPHYSICAL, the least thing I would want to do is remove the logic in rpcrdma_ia_open() that switches to ALLPHYSICAL if the mode selected by rdma_memreg_strategy isn’t supported by the HCA. -- Chuck Lever chuck[dot]lever[at]oracle[dot]com ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-03-01 2:59 ` Chuck Lever @ 2014-03-01 21:29 ` Tom Tucker -1 siblings, 0 replies; 26+ messages in thread From: Tom Tucker @ 2014-03-01 21:29 UTC (permalink / raw) To: Chuck Lever, Wendy Cheng Cc: Tom Talpey, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma Hi Chuck, I have a patch for the server side that simplifies the memory registration and fixes a bug where the server ignores the FRMR hardware limits. This bug is actually upstream now. I have been sitting on it because it's a big patch and will require a lot of testing/review to get it upstream. This is Just an FYI in case there is someone on your team who has the bandwidth to take this work and finish it up. Thanks, Tom On 2/28/14 8:59 PM, Chuck Lever wrote: > Hi Wendy- > > On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8@public.gmane.orgm> wrote: >>> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote: >>>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>>> Hi- >>>>> >>>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>>> the Linux kernel. So far we've built and run functional tests to determine >>>>> what is working and what is broken. >>>>> >>>>> [snip] >>> >>>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>>> All HCAs support this mode. >>>> >>>> Not safe is an understatement. It exposes all of client physical >>>> memory to the peer, for both read and write. A simple pointer error >>>> on the server will silently corrupt the client. This mode was >>>> intended only for testing, and in experimental deployments. >> (sorry, resend .. previous reply bounced back due to gmail html format) >> >> Please keep "ALLPHYSICAL" for now - as our embedded system needs it. > This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. > > Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? > > -- > Chuck Lever > chuck[dot]lever[at]oracle[dot]com > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-01 21:29 ` Tom Tucker 0 siblings, 0 replies; 26+ messages in thread From: Tom Tucker @ 2014-03-01 21:29 UTC (permalink / raw) To: Chuck Lever, Wendy Cheng Cc: Tom Talpey, linux-rdma, Linux NFS Mailing List, Shirley Ma Hi Chuck, I have a patch for the server side that simplifies the memory registration and fixes a bug where the server ignores the FRMR hardware limits. This bug is actually upstream now. I have been sitting on it because it's a big patch and will require a lot of testing/review to get it upstream. This is Just an FYI in case there is someone on your team who has the bandwidth to take this work and finish it up. Thanks, Tom On 2/28/14 8:59 PM, Chuck Lever wrote: > Hi Wendy- > > On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: > >> On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >>> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom@talpey.com> wrote: >>>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>>> Hi- >>>>> >>>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>>> the Linux kernel. So far we've built and run functional tests to determine >>>>> what is working and what is broken. >>>>> >>>>> [snip] >>> >>>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>>> All HCAs support this mode. >>>> >>>> Not safe is an understatement. It exposes all of client physical >>>> memory to the peer, for both read and write. A simple pointer error >>>> on the server will silently corrupt the client. This mode was >>>> intended only for testing, and in experimental deployments. >> (sorry, resend .. previous reply bounced back due to gmail html format) >> >> Please keep "ALLPHYSICAL" for now - as our embedded system needs it. > This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. > > Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? > > -- > Chuck Lever > chuck[dot]lever[at]oracle[dot]com > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <53125125.9010709-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-03-01 21:29 ` Tom Tucker @ 2014-03-02 22:28 ` Chuck Lever -1 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-02 22:28 UTC (permalink / raw) To: Tom Tucker Cc: Wendy Cheng, Tom Talpey, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma On Mar 1, 2014, at 4:29 PM, Tom Tucker <tom-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> wrote: > Hi Chuck, > > I have a patch for the server side that simplifies the memory registration and fixes a bug where the server ignores the FRMR hardware limits. This bug is actually upstream now. > > I have been sitting on it because it's a big patch and will require a lot of testing/review to get it upstream. This is Just an FYI in case there is someone on your team who has the bandwidth to take this work and finish it up. Why not post what you have, and then we can see what can be done. > > Thanks, > Tom > > On 2/28/14 8:59 PM, Chuck Lever wrote: >> Hi Wendy- >> >> On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> >>> On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >>>> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> wrote: >>>>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>>>> Hi- >>>>>> >>>>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>>>> the Linux kernel. So far we've built and run functional tests to determine >>>>>> what is working and what is broken. >>>>>> >>>>>> [snip] >>>> >>>>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>>>> All HCAs support this mode. >>>>> >>>>> Not safe is an understatement. It exposes all of client physical >>>>> memory to the peer, for both read and write. A simple pointer error >>>>> on the server will silently corrupt the client. This mode was >>>>> intended only for testing, and in experimental deployments. >>> (sorry, resend .. previous reply bounced back due to gmail html format) >>> >>> Please keep "ALLPHYSICAL" for now - as our embedded system needs it. >> This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. >> >> Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? >> >> -- >> Chuck Lever >> chuck[dot]lever[at]oracle[dot]com >> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-02 22:28 ` Chuck Lever 0 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-02 22:28 UTC (permalink / raw) To: Tom Tucker Cc: Wendy Cheng, Tom Talpey, linux-rdma, Linux NFS Mailing List, Shirley Ma On Mar 1, 2014, at 4:29 PM, Tom Tucker <tom@opengridcomputing.com> wrote: > Hi Chuck, > > I have a patch for the server side that simplifies the memory registration and fixes a bug where the server ignores the FRMR hardware limits. This bug is actually upstream now. > > I have been sitting on it because it's a big patch and will require a lot of testing/review to get it upstream. This is Just an FYI in case there is someone on your team who has the bandwidth to take this work and finish it up. Why not post what you have, and then we can see what can be done. > > Thanks, > Tom > > On 2/28/14 8:59 PM, Chuck Lever wrote: >> Hi Wendy- >> >> On Feb 28, 2014, at 5:26 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >> >>> On Fri, Feb 28, 2014 at 2:20 PM, Wendy Cheng <s.wendy.cheng@gmail.com> wrote: >>>> ni i...On Fri, Feb 28, 2014 at 1:41 PM, Tom Talpey <tom@talpey.com> wrote: >>>>> On 2/26/2014 8:44 AM, Chuck Lever wrote: >>>>>> Hi- >>>>>> >>>>>> Shirley Ma and I are reviving work on the NFS/RDMA client code base in >>>>>> the Linux kernel. So far we've built and run functional tests to determine >>>>>> what is working and what is broken. >>>>>> >>>>>> [snip] >>>> >>>>>> ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. >>>>>> All HCAs support this mode. >>>>> >>>>> Not safe is an understatement. It exposes all of client physical >>>>> memory to the peer, for both read and write. A simple pointer error >>>>> on the server will silently corrupt the client. This mode was >>>>> intended only for testing, and in experimental deployments. >>> (sorry, resend .. previous reply bounced back due to gmail html format) >>> >>> Please keep "ALLPHYSICAL" for now - as our embedded system needs it. >> This is just the client side. Confirming that you still need support for the ALLPHYSICAL memory registration mode in the NFS/RDMA client. >> >> Do you have plans to move to a mode that is less risky? If not, can we depend on you to perform regular testing with ALLPHYSICAL as we update the client code? Do you have any bug fixes you’d like to merge upstream? >> >> -- >> Chuck Lever >> chuck[dot]lever[at]oracle[dot]com >> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chuck Lever chuck[dot]lever[at]oracle[dot]com ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-02-26 16:44 ` Chuck Lever @ 2014-03-03 17:02 ` Chuck Lever -1 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-03 17:02 UTC (permalink / raw) To: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List; +Cc: Shirley Ma On Feb 26, 2014, at 11:44 AM, Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote: > Hi- > > Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. > > One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. > > Not all HCAs support all memory registration modes, so our test matrix is quite large. I’d like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can’t test. > > BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I’m told it was never intended for production use. > > REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. > > MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. > > MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. > > MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. > > FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. > > ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. > > > I propose removing BOUNCEBUFFERS since it is not intended for production use. > > I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. > > I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. > > I propose removing MTHCA_FMR as I’m told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. > > This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. > > We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. > > Thoughts/comments? Karma score so far: BOUNCEBUFFERS, REGISTER, MEM_WINDOWS, MEM_WINDOWS_ASYNC - no votes in favor of keeping MTHCA_FMR, FRMR, ALLPHYSICAL - one or more votes in favor of keeping All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy’s HCA supports only ALLPHYSICAL. Does it make sense to deprecate then remove the registration modes in the first list? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-03 17:02 ` Chuck Lever 0 siblings, 0 replies; 26+ messages in thread From: Chuck Lever @ 2014-03-03 17:02 UTC (permalink / raw) To: linux-rdma, Linux NFS Mailing List; +Cc: Shirley Ma On Feb 26, 2014, at 11:44 AM, Chuck Lever <chuck.lever@oracle.com> wrote: > Hi- > > Shirley Ma and I are reviving work on the NFS/RDMA client code base in the Linux kernel. So far we’ve built and run functional tests to determine what is working and what is broken. > > One complication is the number of memory registration modes supported by the RPC/RDMA transport: there are seven. These were added over the years to support particular HCAs or as proof-of-concept. The transport chooses a registration mode at mount time based on what the link HCA supports. > > Not all HCAs support all memory registration modes, so our test matrix is quite large. I’d like to propose removing support for one or more of these memory registration modes in the name of making it easier to change this code and test it without breaking something that we can’t test. > > BOUNCEBUFFERS - All HCAs support this mode. Does not use RDMA READ and WRITE, and the client end copies data into place. RDMA is offloaded, by data copy is not. I’m told it was never intended for production use. > > REGISTER - Safe but relatively slow. Uses reg_phys_mr verb which is not supported in mlx4/mlx5, but all other HCAs/providers can use this mode. > > MEM_WINDOWS - Uses bind_mr verb. Safe, but supports only a narrow range of HCAs. > > MEM_WINDOWS_ASYNC - Not always safe, and only a narrow range of HCAs is supported. > > MTHCA_FMR - Uses alloc_fmr verb. Safe, reasonably fast, but only a narrow range of older HCAs is supported. > > FRMR - Safe, generally fast. Currently the preferred registration mode, but is not supported with some older HCAs/providers. > > ALLPHYSICAL - Usually fast, but not safe as it exposes client memory. All HCAs support this mode. > > > I propose removing BOUNCEBUFFERS since it is not intended for production use. > > I propose removing ALLPHYSICAL and MEM_WINDOWS_ASYNC as they are not generally safe. RFC 5666 suggests that unsafe memory registration modes be avoided. > > I propose removing MEM_WINDOWS as it adds complexity without adding a lot of HCA compatibility. > > I propose removing MTHCA_FMR as I’m told it is hard to obtain HCAs we would need for testing this registration mode, and these are all old adapters anyway. > > This leaves NFS/RDMA client support for REGISTER and FRMR, which should cover all existing HCAs, and it is easy to test both of these memory registration modes with just one or two well-picked HCAs. > > We would contribute these changes to the client code base. The NFS/RDMA server code could use similar attention, but we are not volunteering to change it at this time. > > Thoughts/comments? Karma score so far: BOUNCEBUFFERS, REGISTER, MEM_WINDOWS, MEM_WINDOWS_ASYNC - no votes in favor of keeping MTHCA_FMR, FRMR, ALLPHYSICAL - one or more votes in favor of keeping All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy’s HCA supports only ALLPHYSICAL. Does it make sense to deprecate then remove the registration modes in the first list? -- Chuck Lever chuck[dot]lever[at]oracle[dot]com ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <E755E683-C064-4ED3-8448-C61F4E5AF16E-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-03-03 17:02 ` Chuck Lever @ 2014-03-03 19:07 ` Christoph Hellwig -1 siblings, 0 replies; 26+ messages in thread From: Christoph Hellwig @ 2014-03-03 19:07 UTC (permalink / raw) To: Chuck Lever Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma, Wendy Cheng On Mon, Mar 03, 2014 at 12:02:33PM -0500, Chuck Lever wrote: > All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy?s HCA supports only ALLPHYSICAL. Is Wendy planning to submit her HCA driver ASAP? If not there's not reason to keep ALLPHYSICAL either. > Does it make sense to deprecate then remove the registration modes in the first list? Yes. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-03 19:07 ` Christoph Hellwig 0 siblings, 0 replies; 26+ messages in thread From: Christoph Hellwig @ 2014-03-03 19:07 UTC (permalink / raw) To: Chuck Lever; +Cc: linux-rdma, Linux NFS Mailing List, Shirley Ma, Wendy Cheng On Mon, Mar 03, 2014 at 12:02:33PM -0500, Chuck Lever wrote: > All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy?s HCA supports only ALLPHYSICAL. Is Wendy planning to submit her HCA driver ASAP? If not there's not reason to keep ALLPHYSICAL either. > Does it make sense to deprecate then remove the registration modes in the first list? Yes. ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <20140303190733.GA4556-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-03-03 19:07 ` Christoph Hellwig @ 2014-03-03 19:54 ` faibish, sorin -1 siblings, 0 replies; 26+ messages in thread From: faibish, sorin @ 2014-03-03 19:54 UTC (permalink / raw) To: Christoph Hellwig Cc: Chuck Lever, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma, Wendy Cheng Sent from my iPad > On Mar 3, 2014, at 7:09 PM, "Christoph Hellwig" <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> wrote: > >> On Mon, Mar 03, 2014 at 12:02:33PM -0500, Chuck Lever wrote: >> All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy?s HCA supports only ALLPHYSICAL. > > Is Wendy planning to submit her HCA driver ASAP? If not there's not > reason to keep ALLPHYSICAL either. I second Christoph. Legacy is good as long as there are users of Linux with the legacy server. I would say that the only reason to keep it is if Linux server will support it. Same we apply to Lustre client in kernel. ./Sorin > >> Does it make sense to deprecate then remove the registration modes in the first list? > > Yes. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-03 19:54 ` faibish, sorin 0 siblings, 0 replies; 26+ messages in thread From: faibish, sorin @ 2014-03-03 19:54 UTC (permalink / raw) To: Christoph Hellwig Cc: Chuck Lever, linux-rdma, Linux NFS Mailing List, Shirley Ma, Wendy Cheng Sent from my iPad > On Mar 3, 2014, at 7:09 PM, "Christoph Hellwig" <hch@infradead.org> wrote: > >> On Mon, Mar 03, 2014 at 12:02:33PM -0500, Chuck Lever wrote: >> All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy?s HCA supports only ALLPHYSICAL. > > Is Wendy planning to submit her HCA driver ASAP? If not there's not > reason to keep ALLPHYSICAL either. I second Christoph. Legacy is good as long as there are users of Linux with the legacy server. I would say that the only reason to keep it is if Linux server will support it. Same we apply to Lustre client in kernel. ./Sorin > >> Does it make sense to deprecate then remove the registration modes in the first list? > > Yes. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <79E85B7D-5ECC-4A63-BB07-C45A570DA53D-mb1K0bWo544@public.gmane.org>]
* Re: Proposal for simplifying NFS/RDMA client memory registration 2014-03-03 19:54 ` faibish, sorin @ 2014-03-03 20:33 ` Wendy Cheng -1 siblings, 0 replies; 26+ messages in thread From: Wendy Cheng @ 2014-03-03 20:33 UTC (permalink / raw) To: faibish, sorin Cc: Christoph Hellwig, Chuck Lever, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Linux NFS Mailing List, Shirley Ma, wendy.cheng-ral2JQCrhuEAvxtiuMwx3w On Mon, Mar 3, 2014 at 11:54 AM, faibish, sorin <faibish_sorin-mb1K0bWo544@public.gmane.org> wrote: > > > On Mar 3, 2014, at 7:09 PM, "Christoph Hellwig" <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> wrote: > > > >> On Mon, Mar 03, 2014 at 12:02:33PM -0500, Chuck Lever wrote: > >> All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy?s HCA supports only ALLPHYSICAL. > > > > Is Wendy planning to submit her HCA driver ASAP? If not there's not > > reason to keep ALLPHYSICAL either. > I second Christoph. Legacy is good as long as there are users of Linux with the legacy server. I would say that the only reason to keep it is if Linux server will support it. Same we apply to Lustre client in kernel. > > ./Sorin > > > > >> Does it make sense to deprecate then remove the registration modes in the first list? > > > > Yes. > > > After discussing this with my manager, we'll let it go for now ... will re-submit the full patch set in the future when we finalize the plan. Thanks, Wendy -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Proposal for simplifying NFS/RDMA client memory registration @ 2014-03-03 20:33 ` Wendy Cheng 0 siblings, 0 replies; 26+ messages in thread From: Wendy Cheng @ 2014-03-03 20:33 UTC (permalink / raw) To: faibish, sorin Cc: Christoph Hellwig, Chuck Lever, linux-rdma, Linux NFS Mailing List, Shirley Ma, wendy.cheng On Mon, Mar 3, 2014 at 11:54 AM, faibish, sorin <faibish_sorin@emc.com> wrote: > > > On Mar 3, 2014, at 7:09 PM, "Christoph Hellwig" <hch@infradead.org> wrote: > > > >> On Mon, Mar 03, 2014 at 12:02:33PM -0500, Chuck Lever wrote: > >> All HCAs in 3.13 (and rxe) can support either MTHCA_FMR or FRMR or both. Wendy?s HCA supports only ALLPHYSICAL. > > > > Is Wendy planning to submit her HCA driver ASAP? If not there's not > > reason to keep ALLPHYSICAL either. > I second Christoph. Legacy is good as long as there are users of Linux with the legacy server. I would say that the only reason to keep it is if Linux server will support it. Same we apply to Lustre client in kernel. > > ./Sorin > > > > >> Does it make sense to deprecate then remove the registration modes in the first list? > > > > Yes. > > > After discussing this with my manager, we'll let it go for now ... will re-submit the full patch set in the future when we finalize the plan. Thanks, Wendy ^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2014-03-03 20:33 UTC | newest] Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-02-26 16:44 Proposal for simplifying NFS/RDMA client memory registration Chuck Lever 2014-02-26 16:44 ` Chuck Lever [not found] ` <01C4496A-F074-4F72-9DF0-6076C05E8A1F-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-02-26 17:42 ` faibish, sorin 2014-02-26 17:42 ` faibish, sorin 2014-02-28 21:41 ` Tom Talpey 2014-02-28 21:41 ` Tom Talpey [not found] ` <CABgxfbHwo13ma=-Xn+S36WwD8LVNLdw6UHztFe8EkCA_=NBenw@mail.gmail.com> [not found] ` <CABgxfbHwo13ma=-Xn+S36WwD8LVNLdw6UHztFe8EkCA_=NBenw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2014-02-28 22:26 ` Wendy Cheng 2014-02-28 22:26 ` Wendy Cheng [not found] ` <CABgxfbHg0B02c2zpYdCB4pUZZhhA4aKRuyh4Kx=NE1yAisFLLQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2014-03-01 2:59 ` Chuck Lever 2014-03-01 2:59 ` Chuck Lever [not found] ` <B83B4DA6-E9CF-4DF3-9227-5EF9B4D25F4D-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-03-01 16:00 ` Jeff Layton 2014-03-01 16:00 ` Jeff Layton [not found] ` <20140301110022.417eb088-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org> 2014-03-01 17:14 ` Chuck Lever 2014-03-01 17:14 ` Chuck Lever 2014-03-01 21:29 ` Tom Tucker 2014-03-01 21:29 ` Tom Tucker [not found] ` <53125125.9010709-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> 2014-03-02 22:28 ` Chuck Lever 2014-03-02 22:28 ` Chuck Lever 2014-03-03 17:02 ` Chuck Lever 2014-03-03 17:02 ` Chuck Lever [not found] ` <E755E683-C064-4ED3-8448-C61F4E5AF16E-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-03-03 19:07 ` Christoph Hellwig 2014-03-03 19:07 ` Christoph Hellwig [not found] ` <20140303190733.GA4556-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2014-03-03 19:54 ` faibish, sorin 2014-03-03 19:54 ` faibish, sorin [not found] ` <79E85B7D-5ECC-4A63-BB07-C45A570DA53D-mb1K0bWo544@public.gmane.org> 2014-03-03 20:33 ` Wendy Cheng 2014-03-03 20:33 ` Wendy Cheng
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.