mountd gives "illegal port"

mountd gives "illegal port"

[Jan Kasprzak]

	Hi all,

I have discovered the following problem in Linux NFS: When you mount
many filesystems from the same server, mountd sometimes refuses the
request with "illegal port" message in syslog. To reproduce this, I did
the following:

On the NFS server:
# for i in `seq 1 2048`; do mkdir /tmp/nfs/$i; echo "$i" >/tmp/nfs/$i/index-$i.txt; echo "/tmp/nfs/$i    *.fi.muni.cz(ro)"; done >/etc/exports
# exportfs -a

On the NFS client;
# for i in `seq 1 2048`; do mkdir /tmp/nfs/$i; done
# for i in `seq 1 100`; do echo $i; mount -t nfs -r nfs-server:/tmp/nfs/$i /tmp/nfs/$i || break; done

It fails after random number of iterations (seen 6 to 57) with

mount: nfs-server:/tmp/nfs/6 failed, reason given by server: Permission denied

And in NFS server the following is added to syslog:

Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal port 54071

The situation seems a bit better when you add "sleep 5" to the mount loop.
Sleep 2 does not help, though.

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[J. Bruce Fields]

On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
> I have discovered the following problem in Linux NFS: When you mount
> many filesystems from the same server, mountd sometimes refuses the
> request with "illegal port" message in syslog. To reproduce this, I did
> the following:

Just add the "insecure" export option to the server's export.

--Bruce Fields


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[Jan Kasprzak]

Jan Kasprzak wrote:
: I have discovered the following problem in Linux NFS: When you mount
: many filesystems from the same server, mountd sometimes refuses the
: request with "illegal port" message in syslog.

	Well, the mountd is probably OK, and it is the client who runs
out of privileged ports. netstat -ta on the client looks like this:

tcp        0      0 nfs-client.fi.muni.cz:880  nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp        0      0 nfs-client.fi.muni.cz:883  nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp        0      0 nfs-client.fi.muni.cz:884  nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp        0      0 nfs-client.fi.muni.cz:887  nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp        0      0 nfs-client.fi.muni.cz:888  nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp        0      0 nfs-client.fi.muni.cz:891  nfs-server.fi.muni.c:sunrpc TIME_WAIT
[...]    

	The solution might be to query portmap over UDP from the NFS client
instead of TCP. Or to query portmap from an unprivileged port (why we
send query to portmap from privileged port anyway?).

	 I have even tried echo 2 > /proc/sys/net/ipv4/tcp_fin_timeout,
but this did not solve the problem - connections to server's portmap
remain for 60 seconds in FIN_WAIT.

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[Jan Kasprzak]

J. Bruce Fields wrote:
: On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
: > I have discovered the following problem in Linux NFS: When you mount
: > many filesystems from the same server, mountd sometimes refuses the
: > request with "illegal port" message in syslog. To reproduce this, I did
: > the following:
: 
: Just add the "insecure" export option to the server's export.
: 
	Of course, but I don't want to do this. The problem is that client
is querying portmap over TCP and from privileged port. By changing this
we can use twice as much mounts in the TIME_WAIT period. We may also
force mount to query mountd over UDP, but I don't know how to do this.
-o udp is probably for nfsd traffic, isn't it?

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[J. Bruce Fields]

On Wed, Oct 27, 2004 at 06:23:22PM +0200, Jan Kasprzak wrote:
> J. Bruce Fields wrote:
> : On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
> : > I have discovered the following problem in Linux NFS: When you mount
> : > many filesystems from the same server, mountd sometimes refuses the
> : > request with "illegal port" message in syslog. To reproduce this, I did
> : > the following:
> : 
> : Just add the "insecure" export option to the server's export.
> : 
> 	Of course, but I don't want to do this.

Why not?

> The problem is that client is querying portmap over TCP and from
> privileged port. By changing this we can use twice as much mounts in
> the TIME_WAIT period.

OK.  It seems like kind of an obscure thing to care about....

> We may also force mount to query mountd over UDP, but I don't know how
> to do this.  -o udp is probably for nfsd traffic, isn't it?

That's what I'd assume, but I don't know.--b.


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[Jan Kasprzak]

J. Bruce Fields wrote:
: > : 
: > : Just add the "insecure" export option to the server's export.
: > : 
: > 	Of course, but I don't want to do this.
: 
: Why not?
: 
	NFS client is a host where ordinary (=evil :-) users have accounts.

-Y.

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[Peter Astrand]

On Wed, 27 Oct 2004, Jan Kasprzak wrote:

> 	Of course, but I don't want to do this. The problem is that client
> is querying portmap over TCP and from privileged port. By changing this
> we can use twice as much mounts in the TIME_WAIT period. We may also
> force mount to query mountd over UDP, but I don't know how to do this.
> -o udp is probably for nfsd traffic, isn't it?

As far as I know, this has never been documented. This is somewhat related
to the Fedora bug 128100: Even if you specify -o tcp, port= and
mountport=, the mount command tries to "autodetect" if it should use TCP
or UDP for the MOUNT protocol by contacting the portmapper (which fails in 
many cases, if you use SSH tunnels, for example). 

Most people nowadays have realized that it was a mistake to separate the 
NFS and the MOUNT protocol. It's much more convenient to run them on the 
same port, using the same transport. With the current mountd/knfsd 
implementation it might be a little hard to use the same port number, but 
with unfsd, for example, it's easy to use the same port. Using the same 
transport type, though, is easy. 
 
So, my opinion is that you should either use TCP for both the NFS and 
MOUNT protocol, or UDP for both. 

Perhaps this "illegal port" problem can be solved by making the mount 
command use SO_REUSEADDR?


-- 
Peter Åstrand		Chief Developer
Cendio			www.thinlinc.com
Teknikringen 3		www.cendio.se
583 30 Linköping        Phone: +46-13-21 46 00



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[Michael Richardson]

-----BEGIN PGP SIGNED MESSAGE-----


>>>>> "Jan" == Jan Kasprzak <kas@fi.muni.cz> writes:
    Jan> I have discovered the following problem in Linux NFS: When you

  What version are you using?
  On client? on server?

    Jan> And in NFS server the following is added to syslog:

    Jan> Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request
    Jan> from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal
    Jan> port 54071

  Does your server expect to see reserved ports only (<1024).
  Can you do a tcpdump of the mounts?

  To do this, on the server do:

     rpcinfo -p localhost

  note the port number of the mountd service. Let's say it is X.

    tcpdump -s 1600 -i eth0 -n -p -w somefile.pcap udp port X or tcp port X


  tcpdump -r somefile.pcap -T rpc -n 

- --
]     "Elmo went to the wrong fundraiser" - The Simpson         |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] mcr@xelerance.com      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys

iQCVAwUBQYANXoqHRg3pndX9AQGncAP/bj6qf9FcHKdu0I37JGPzisrdeiI9VBV7
wYjwONqRjfyhub+STaM5Upid8uwpzSw3rs1ixZJeXX5PklSmDz5XW0Gik/oauwWL
OfPik8NEUsZNjSnS6sjaLCRRUC7PKAB3wTap6IMq/cskLjBTACJ6Gzj2vNYu9sSD
96BPplHUrcs=
=liY1
-----END PGP SIGNATURE-----


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[Jan Kasprzak]

Michael Richardson wrote:
:   What version are you using?
:   On client? on server?
: 
	Kernel 2.6.9 on both hosts (server is x86-64, client i386),
client nfs-utils-1.0.6-22 on Fedora Core 2, server nfs-utils-1.0.6-34
on Fedora Core 3 test 3.

:     Jan> And in NFS server the following is added to syslog:
: 
:     Jan> Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request
:     Jan> from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal
:     Jan> port 54071
: 
:   Does your server expect to see reserved ports only (<1024).

	Yes (there is no "insecure" option in /etc/exports).

:   Can you do a tcpdump of the mounts?
: 
	Yes - I did. The failing mount request uses unprivileged source port
unlike the previous ones.

:     tcpdump -s 1600 -i eth0 -n -p -w somefile.pcap udp port X or tcp port X
: 
: 
:   tcpdump -r somefile.pcap -T rpc -n 

	I will do this on Monday - now I am not nowhere near these servers.

	But as I said, the problem is that the client runs out of privileged
ports, because both portmap request and mountd request are done over TCP.

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while.      --Rob Pike <


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: mountd gives "illegal port"

[Michael Richardson]

-----BEGIN PGP SIGNED MESSAGE-----


>>>>> "Jan" == Jan Kasprzak <kas@fi.muni.cz> writes:
    Jan> :   Does your server expect to see reserved ports only (<1024).

    Jan> Yes (there is no "insecure" option in /etc/exports).
 
  Yes, the rest of the thread arrived.
  
  Clearly, you are running out of reserved ports.

- --
]     "Elmo went to the wrong fundraiser" - The Simpson         |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] mcr@xelerance.com      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys

iQCVAwUBQYAe1IqHRg3pndX9AQHbNwP9FAC+myGfExU3Cjl4phvRW48fpOUkOYci
80blUbA/Br+qOQvX+MMX52rSN0z7laQRV00LK7i1QXkk/uVUuwfIrxPVuZYIFm0S
OSXHXA8jp6bH9ypcBivd9B4t1FvyLFb199VsyoMkav0JeJ1tornCQBKDK0TECfm5
yX6BwrP579k=
=FsWx
-----END PGP SIGNATURE-----


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs