RE: Re: [autofs] multiple servers per automount

RE: Re: [autofs] multiple servers per automount

[Lever, Charles]

the problem is likely the algorithm used to allocate
ports for the RPC transport sockets.  it starts at
port 800 and goes down to zero.

> -----Original Message-----
> From: Ian Kent [mailto:raven@themaw.net]
> Sent: Thursday, October 09, 2003 6:09 PM
> To: Mike Waychison
> Cc: Ogden, Aaron A.; autofs mailing list; nfs@lists.sourceforge.net
> Subject: [NFS] Re: [autofs] multiple servers per automount
>=20
>=20
> On Thu, 9 Oct 2003, Mike Waychison wrote:
>=20
> > Ogden, Aaron A. wrote:
> >
> > >Ouch.  As you may know, the limit is *much* lower in=20
> linux.  Something
> > >that I've been struggling with recently...
> > >
> > >Under normal circumstances I would not be concerned with=20
> 'limitations'
> > >of a few hundred active NFS mounts, but such limitations=20
> certainly limit
> > >scalability for the extreme cases.
> > >
> > >
> >
> > The maximum number of plain pseudo-block device filesystems=20
> on a given
> > filesystem is limitted to 256. (This includes proc, autofs, nfs..).
> >
> > This is because pseudo-block filesystems all use major 0,=20
> and each have
> > a different minor (thus the 256 limit).
> >
> > There are however patches floating around (look at SuSe's=20
> kernels, I'm
> > not sure about RH) that allow n majors to be used (default 5).  This
> > gives you 1280 mounts, a big step up :)
> >
>=20
> But as Aaron and I know things go pear shaped at just shy of=20
> 800 mounts
> with RedHat kernels. They have the more-unnamed patch.
>=20
> So this would indicate that even if there is a device system that can
> increase the number of unnamed devices that subsystems like NFS cannot
> handle this many mounts.
>=20
> --=20
>=20
>    ,-._|\    Ian Kent
>   /      \   Perth, Western Australia
>   *_.--._/   E-mail: raven@themaw.net
>         v    Web: http://themaw.net/
>=20
>=20
>=20
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects.
> See the people who have HELPED US provide better services:
> Click here: http://sourceforge.net/supporters.php
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
>=20


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: [autofs] multiple servers per automount

[Lever, Charles]

the problem is likely the algorithm used to allocate
ports for the RPC transport sockets.  it starts at
port 800 and goes down to zero.

> -----Original Message-----
> From: Ian Kent [mailto:raven@themaw.net]
> Sent: Thursday, October 09, 2003 6:09 PM
> To: Mike Waychison
> Cc: Ogden, Aaron A.; autofs mailing list; nfs@lists.sourceforge.net
> Subject: [NFS] Re: [autofs] multiple servers per automount
> 
> 
> On Thu, 9 Oct 2003, Mike Waychison wrote:
> 
> > Ogden, Aaron A. wrote:
> >
> > >Ouch.  As you may know, the limit is *much* lower in 
> linux.  Something
> > >that I've been struggling with recently...
> > >
> > >Under normal circumstances I would not be concerned with 
> 'limitations'
> > >of a few hundred active NFS mounts, but such limitations 
> certainly limit
> > >scalability for the extreme cases.
> > >
> > >
> >
> > The maximum number of plain pseudo-block device filesystems 
> on a given
> > filesystem is limitted to 256. (This includes proc, autofs, nfs..).
> >
> > This is because pseudo-block filesystems all use major 0, 
> and each have
> > a different minor (thus the 256 limit).
> >
> > There are however patches floating around (look at SuSe's 
> kernels, I'm
> > not sure about RH) that allow n majors to be used (default 5).  This
> > gives you 1280 mounts, a big step up :)
> >
> 
> But as Aaron and I know things go pear shaped at just shy of 
> 800 mounts
> with RedHat kernels. They have the more-unnamed patch.
> 
> So this would indicate that even if there is a device system that can
> increase the number of unnamed devices that subsystems like NFS cannot
> handle this many mounts.
> 
> -- 
> 
>    ,-._|\    Ian Kent
>   /      \   Perth, Western Australia
>   *_.--._/   E-mail: raven@themaw.net
>         v    Web: http://themaw.net/
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> SourceForge.net hosts over 70,000 Open Source Projects.
> See the people who have HELPED US provide better services:
> Click here: http://sourceforge.net/supporters.php
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> 


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [NFS] Re: multiple servers per automount

[Ian Kent]

On Fri, 10 Oct 2003, Lever, Charles wrote:

> the problem is likely the algorithm used to allocate
> ports for the RPC transport sockets.  it starts at
> port 800 and goes down to zero.

Don't think so.

I appears that a single connection is maintained for nfs comms for both
udp and tcp.

However, if a rapid number of mount requests are fired then multiple
portmap connections are made. They end up in a TIME_WAIT state which is
probably causing the port allocation starvation.

This doesn't appear to happen under Solaris.

>
> > -----Original Message-----
> > From: Ian Kent [mailto:raven@themaw.net]
> > Sent: Thursday, October 09, 2003 6:09 PM
> > To: Mike Waychison
> > Cc: Ogden, Aaron A.; autofs mailing list; nfs@lists.sourceforge.net
> > Subject: [NFS] Re: [autofs] multiple servers per automount
> >
> >
> > On Thu, 9 Oct 2003, Mike Waychison wrote:
> >
> > > Ogden, Aaron A. wrote:
> > >
> > > >Ouch.  As you may know, the limit is *much* lower in
> > linux.  Something
> > > >that I've been struggling with recently...
> > > >
> > > >Under normal circumstances I would not be concerned with
> > 'limitations'
> > > >of a few hundred active NFS mounts, but such limitations
> > certainly limit
> > > >scalability for the extreme cases.
> > > >
> > > >
> > >
> > > The maximum number of plain pseudo-block device filesystems
> > on a given
> > > filesystem is limitted to 256. (This includes proc, autofs, nfs..).
> > >
> > > This is because pseudo-block filesystems all use major 0,
> > and each have
> > > a different minor (thus the 256 limit).
> > >
> > > There are however patches floating around (look at SuSe's
> > kernels, I'm
> > > not sure about RH) that allow n majors to be used (default 5).  This
> > > gives you 1280 mounts, a big step up :)
> > >
> >
> > But as Aaron and I know things go pear shaped at just shy of
> > 800 mounts
> > with RedHat kernels. They have the more-unnamed patch.
> >
> > So this would indicate that even if there is a device system that can
> > increase the number of unnamed devices that subsystems like NFS cannot
> > handle this many mounts.
> >
> > --
> >
> >    ,-._|\    Ian Kent
> >   /      \   Perth, Western Australia
> >   *_.--._/   E-mail: raven@themaw.net
> >         v    Web: http://themaw.net/
> >
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: SF.net Giveback Program.
> > SourceForge.net hosts over 70,000 Open Source Projects.
> > See the people who have HELPED US provide better services:
> > Click here: http://sourceforge.net/supporters.php
> > _______________________________________________
> > NFS maillist  -  NFS@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs
> >
>

-- 

   ,-._|\    Ian Kent
  /      \   Perth, Western Australia
  *_.--._/   E-mail: raven@themaw.net
        v    Web: http://themaw.net/

Re: RE: [autofs] multiple servers per automount

[Ian Kent]

On Tue, 14 Oct 2003, Mike Waychison wrote:

> Ian Kent wrote:
>
> >On Tue, 14 Oct 2003, Joseph V Moss wrote:
> >
> >
> >
> >>The limit is 800 as others have stated.  Although, it can be less than that
> >>if something else is already using up some of the reserved UDP ports.
> >>
> >>I wrote a patch long ago against a 2.2.x kernel to enable it to use
> >>multiple majors for NFS mounts (like the patches now common in several
> >>distros).  I then ran into the 800 limit in the RPC layer.  After changing
> >>the RPC layer to count up from 0, instead of down from 800, with no real
> >>upper limit, I was able to mount more than 2000 NFS filesystems simultaneously.
> >>I'm sure I could have done many thousand if I had had that many filesystems
> >>around to mount.  Obviously, after 1024, it wasn't using reserved ports
> >>anymore, but it didn't seem to matter.
> >>
> >>Unfortunately, while the changes to NFS were easy to port to the 2.4 kernel,
> >>the RPC layer is different enough between 2.2 and 2.4 that it didn't work
> >>right off.  Bumping it up to somewhere around 1024 should work, but using
> >>non-reserved ports didn't seem to work when I made a simple attempt.
> >>
> >>Of course, the real fix for the NFS layer is the expansion of the minor
> >>numbers that's already occurred in 2.6 and the RPC layer problems should
> >>be fixed by multiplexing multiple mounts on the same port.
> >>
> >>
> >>
> >>
> >
> >I don't see that expansion in 2.6 (test6). It looks to me like the
> >allocation is done in set_anon_super (in fs/super.c) and that looks like
> >it is restricted to 256. Please correct this for me. I can't see how there
> >is any change to the number of unnmaed devices.
> >
> >
> >
>
> Here is the quick fix for this in RH 2.1AS kernels:
>
> http://www.kernelnewbies.org/kernels/rh21as/SOURCES/linux-2.4.9-moreunnamed.patch
>
> It makes unnamed block devices use majors 12, 14, 38, 39, as well as 0.
>
> I don't know if anyone is working out a better scheme for
> get_unnamed_dev in 2.6 yet.  It does need to be done though.  A simple
> patch for 2.6 would maybe see the unnamed_dev_in_use bitmap grow to
> PAGE_SIZE, automatically allowing for 32768 unnamed devices.
>

OK. Sounds like a good job for me to do (simple - maybe).
I'll spend a while looking for possible side effects.

Do you think that the possible NFS port allocation problems should hold up
this work or should it drive updates to NFS?

Comments from anyone about where to check and what to watch out for are
welcome.

-- 

   ,-._|\    Ian Kent
  /      \   Perth, Western Australia
  *_.--._/   E-mail: raven@themaw.net
        v    Web: http://themaw.net/



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: RE: [autofs] multiple servers per automount

[Ian Kent]

On Tue, 14 Oct 2003, Mike Waychison wrote:

> Ian Kent wrote:
>
> >On Tue, 14 Oct 2003, Joseph V Moss wrote:
> >
> >
> >
> >>The limit is 800 as others have stated.  Although, it can be less than that
> >>if something else is already using up some of the reserved UDP ports.
> >>
> >>I wrote a patch long ago against a 2.2.x kernel to enable it to use
> >>multiple majors for NFS mounts (like the patches now common in several
> >>distros).  I then ran into the 800 limit in the RPC layer.  After changing
> >>the RPC layer to count up from 0, instead of down from 800, with no real
> >>upper limit, I was able to mount more than 2000 NFS filesystems simultaneously.
> >>I'm sure I could have done many thousand if I had had that many filesystems
> >>around to mount.  Obviously, after 1024, it wasn't using reserved ports
> >>anymore, but it didn't seem to matter.
> >>
> >>Unfortunately, while the changes to NFS were easy to port to the 2.4 kernel,
> >>the RPC layer is different enough between 2.2 and 2.4 that it didn't work
> >>right off.  Bumping it up to somewhere around 1024 should work, but using
> >>non-reserved ports didn't seem to work when I made a simple attempt.
> >>
> >>Of course, the real fix for the NFS layer is the expansion of the minor
> >>numbers that's already occurred in 2.6 and the RPC layer problems should
> >>be fixed by multiplexing multiple mounts on the same port.
> >>
> >>
> >>
> >>
> >
> >I don't see that expansion in 2.6 (test6). It looks to me like the
> >allocation is done in set_anon_super (in fs/super.c) and that looks like
> >it is restricted to 256. Please correct this for me. I can't see how there
> >is any change to the number of unnmaed devices.
> >
> >
> >
>
> Here is the quick fix for this in RH 2.1AS kernels:
>
> http://www.kernelnewbies.org/kernels/rh21as/SOURCES/linux-2.4.9-moreunnamed.patch
>
> It makes unnamed block devices use majors 12, 14, 38, 39, as well as 0.
>
> I don't know if anyone is working out a better scheme for
> get_unnamed_dev in 2.6 yet.  It does need to be done though.  A simple
> patch for 2.6 would maybe see the unnamed_dev_in_use bitmap grow to
> PAGE_SIZE, automatically allowing for 32768 unnamed devices.
>

OK. Sounds like a good job for me to do (simple - maybe).
I'll spend a while looking for possible side effects.

Do you think that the possible NFS port allocation problems should hold up
this work or should it drive updates to NFS?

Comments from anyone about where to check and what to watch out for are
welcome.

-- 

   ,-._|\    Ian Kent
  /      \   Perth, Western Australia
  *_.--._/   E-mail: raven@themaw.net
        v    Web: http://themaw.net/



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: RE: [autofs] multiple servers per automount

[Ian Kent]

On Tue, 14 Oct 2003, Joseph V Moss wrote:

> The limit is 800 as others have stated.  Although, it can be less than that
> if something else is already using up some of the reserved UDP ports.
> 
> I wrote a patch long ago against a 2.2.x kernel to enable it to use
> multiple majors for NFS mounts (like the patches now common in several
> distros).  I then ran into the 800 limit in the RPC layer.  After changing
> the RPC layer to count up from 0, instead of down from 800, with no real
> upper limit, I was able to mount more than 2000 NFS filesystems simultaneously.
> I'm sure I could have done many thousand if I had had that many filesystems
> around to mount.  Obviously, after 1024, it wasn't using reserved ports
> anymore, but it didn't seem to matter.
> 
> Unfortunately, while the changes to NFS were easy to port to the 2.4 kernel,
> the RPC layer is different enough between 2.2 and 2.4 that it didn't work
> right off.  Bumping it up to somewhere around 1024 should work, but using
> non-reserved ports didn't seem to work when I made a simple attempt.
> 
> Of course, the real fix for the NFS layer is the expansion of the minor
> numbers that's already occurred in 2.6 and the RPC layer problems should
> be fixed by multiplexing multiple mounts on the same port.
> 
> 

I don't see that expansion in 2.6 (test6). It looks to me like the 
allocation is done in set_anon_super (in fs/super.c) and that looks like 
it is restricted to 256. Please correct this for me. I can't see how there 
is any change to the number of unnmaed devices.

-- 

   ,-._|\    Ian Kent
  /      \   Perth, Western Australia
  *_.--._/   E-mail: raven@themaw.net
        v    Web: http://themaw.net/



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: RE: [autofs] multiple servers per automount

[Ian Kent]

On Tue, 14 Oct 2003, Joseph V Moss wrote:

> The limit is 800 as others have stated.  Although, it can be less than that
> if something else is already using up some of the reserved UDP ports.
> 
> I wrote a patch long ago against a 2.2.x kernel to enable it to use
> multiple majors for NFS mounts (like the patches now common in several
> distros).  I then ran into the 800 limit in the RPC layer.  After changing
> the RPC layer to count up from 0, instead of down from 800, with no real
> upper limit, I was able to mount more than 2000 NFS filesystems simultaneously.
> I'm sure I could have done many thousand if I had had that many filesystems
> around to mount.  Obviously, after 1024, it wasn't using reserved ports
> anymore, but it didn't seem to matter.
> 
> Unfortunately, while the changes to NFS were easy to port to the 2.4 kernel,
> the RPC layer is different enough between 2.2 and 2.4 that it didn't work
> right off.  Bumping it up to somewhere around 1024 should work, but using
> non-reserved ports didn't seem to work when I made a simple attempt.
> 
> Of course, the real fix for the NFS layer is the expansion of the minor
> numbers that's already occurred in 2.6 and the RPC layer problems should
> be fixed by multiplexing multiple mounts on the same port.
> 
> 

I don't see that expansion in 2.6 (test6). It looks to me like the 
allocation is done in set_anon_super (in fs/super.c) and that looks like 
it is restricted to 256. Please correct this for me. I can't see how there 
is any change to the number of unnmaed devices.

-- 

   ,-._|\    Ian Kent
  /      \   Perth, Western Australia
  *_.--._/   E-mail: raven@themaw.net
        v    Web: http://themaw.net/



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs