RPC UDP message size limit

RPC UDP message size limit

[Robbert Eggermont]

Recently we started hitting the libtirpc RPC UDP message size limit 
(tirpc/rpc/rpc.h: #define UDPMSGSIZE 8800) for 'showmount -e' from a 
NetApp filer (with around 25kB worth of exports). Only 8800 bytes are 
read, resulting in an abort with a "rpc mount export: RPC: Can't decode 
result" error.

Since it seems filers are already using a higher limit, should this 
limit be increased (at least for receiving, which should be safe)?

-- 
Robbert Eggermont
Intelligent Systems Support & Data Steward | TU Delft
+31 15 27 83234 | Building 28, Floor 5, Room W660
Available Mon, Wed-Fri

Re: RPC UDP message size limit

[Chuck Lever]

> On Jun 1, 2018, at 8:07 AM, Robbert Eggermont <R.Eggermont@tudelft.nl> =
wrote:
>=20
> Recently we started hitting the libtirpc RPC UDP message size limit =
(tirpc/rpc/rpc.h: #define UDPMSGSIZE 8800) for 'showmount -e' from a =
NetApp filer (with around 25kB worth of exports). Only 8800 bytes are =
read, resulting in an abort with a "rpc mount export: RPC: Can't decode =
result" error.

8KB seems to be a documented limit for UDP. =46rom "man clnt_create":

     CLIENT *
     clnt_create(char *host, u_long prog, u_long vers, char *proto)

             Generic client creation routine.  The host argument =
identifies the name of the remote host where the server is located.  The
             proto argument indicates which kind of transport protocol =
to use.  The currently supported values for this field are "udp" and
             "tcp".  Default timeouts are set, but can be modified using =
clnt_control().

             Warning: Using UDP has its shortcomings.  Since UDP-based =
RPC messages can only hold up to 8 Kbytes of encoded data, this
             transport cannot be used for procedures that take large =
arguments or return huge results.

A larger message size limit would not be safe for all RPC protocols
that use UDP.


> Since it seems filers are already using a higher limit, should this =
limit be increased (at least for receiving, which should be safe)?

=46rom utils/showmount/showmount.c

107         client =3D clnt_create(hostname, program, vers, "tcp");
108         if (client)
109                 return client;
110         client =3D clnt_create(hostname, program, vers, "udp");
111         if (client)
112                 return client;

IIUC showmount could use the clnt_{vc,dg}_create(3) APIs instead of
the generic clnt_create(3) API to specify a larger message size limit.

However, using TCP would immediately work around the UDP message size
limit. But perhaps the filer is not allowing a TCP connection for the
MNT service?


--
Chuck Lever
chucklever@gmail.com

Re: RPC UDP message size limit

[Robbert Eggermont]

> 8KB seems to be a documented limit for UDP.

I tried to find documentation for this limit, but all I found was one 
reference to SunOS (of course), because of incompatible network hardware.

The real limit (based on the 16-bit length field defined in RFC 768, 
1980) is 65507 bytes (plus headers, for UDP over IPv4).

> However, using TCP would immediately work around the UDP message size
> limit. But perhaps the filer is not allowing a TCP connection for the
> MNT service?

TCP is not working for us at this moment (because of firewalling), so 
UDP is used as the fallback (as designed).

In our case, I'ld rather have the fallback fail because of a real 
hardware incompatibility than of an artificial software limit... ;-)

-- 
Robbert Eggermont
Intelligent Systems Support & Data Steward | TU Delft
+31 15 27 83234 | Building 28, Floor 5, Room W660
Available Mon, Wed-Fri

Re: RPC UDP message size limit

[Chuck Lever]

> On Jun 4, 2018, at 11:26 AM, Robbert Eggermont =
<R.Eggermont@tudelft.nl> wrote:
>=20
>> 8KB seems to be a documented limit for UDP.
>=20
> I tried to find documentation for this limit, but all I found was one =
reference to SunOS (of course), because of incompatible network =
hardware.

The text I quoted you is from the Linux man page for clnt_create(3).


> The real limit (based on the 16-bit length field defined in RFC 768, =
1980) is 65507 bytes (plus headers, for UDP over IPv4).

That's the architectural limit. There's also a practical limit
that is much lower.

UDP datagrams have to be re-assembled from smaller frames. The IP
layer does that using a 16-bit ID header field and verifies the
re-assembly with a weak checksum. It's pretty easy to fool the
checksum by assembling the frames out of order, which can happen
if the ID header value wraps.

There are other problems that make UDP problematic for any RPC
program that relies on the datagram to contain the correct data
in the correct order.

To mitigate these problems, the maximum size of UDP frames is
reduced. IMO 8KB + a handful is about as large as we ever want to
go in the general case.


>> However, using TCP would immediately work around the UDP message size
>> limit. But perhaps the filer is not allowing a TCP connection for the
>> MNT service?
>=20
> TCP is not working for us at this moment (because of firewalling), so =
UDP is used as the fallback (as designed).
>=20
> In our case, I'ld rather have the fallback fail because of a real =
hardware incompatibility than of an artificial software limit... ;-)

Making a library change that impacts all RPC consumers is IMO
quite inappropriate, for the reason explained above. The
preferred way to address this is to increase the UDP message
size limit only for showmount.


>> IIUC showmount could use the clnt_{vc,dg}_create(3) APIs instead of
>> the generic clnt_create(3) API to specify a larger message size =
limit.


Can you try this suggestion and let us know if it's effective?


--
Chuck Lever
chucklever@gmail.com