All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/3] gssd_proc: use pollsize, not FD_ALLOC_BLOCK, in get_poll_index().
  2012-11-28  1:11 [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Neil Brown
@ 2012-11-28  1:11 ` Neil Brown
  2012-11-28  1:11 ` [PATCH 2/3] gssd_proc: remove pointless test against FD_ALLOC_BLOCK in process_pipedir Neil Brown
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 11+ messages in thread
From: Neil Brown @ 2012-11-28  1:11 UTC (permalink / raw)
  To: Steve Dickson; +Cc: linux-nfs

get_poll_index wants to walk the entire "pollarray", but uses
the constant FD_ALLOC_BLOCK, rather than the variable
pollsize (which has the same value).  If we want to make the
size of the array variable, it is best not to use the constant.

As pollsize is 'unsigned long', 'i' should be too.

Signed-off-by: NeilBrown <neilb@suse.de>
---
 utils/gssd/gssd_proc.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/utils/gssd/gssd_proc.c b/utils/gssd/gssd_proc.c
index ec251fa..2d7ec94 100644
--- a/utils/gssd/gssd_proc.c
+++ b/utils/gssd/gssd_proc.c
@@ -398,10 +398,10 @@ process_clnt_dir_files(struct clnt_info * clp)
 static int
 get_poll_index(int *ind)
 {
-	int i;
+	unsigned int i;
 
 	*ind = -1;
-	for (i=0; i<FD_ALLOC_BLOCK; i++) {
+	for (i=0; i<pollsize; i++) {
 		if (pollarray[i].events == 0) {
 			*ind = i;
 			break;



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic
@ 2012-11-28  1:11 Neil Brown
  2012-11-28  1:11 ` [PATCH 1/3] gssd_proc: use pollsize, not FD_ALLOC_BLOCK, in get_poll_index() Neil Brown
                   ` (3 more replies)
  0 siblings, 4 replies; 11+ messages in thread
From: Neil Brown @ 2012-11-28  1:11 UTC (permalink / raw)
  To: Steve Dickson; +Cc: linux-nfs

We recently had a request to raise the number of fds that gssd can
work with - again.

Rather than just change the hard-coded number I seemed to make sense
to have it depend on RLIMIT_NOFILE - as poll won't work with more than
that anyway.
So the default becomes 1024, but it is easily raised further without
code change.

These patches implement that.

NeilBrown


---

Neil Brown (3):
      gssd_proc: use pollsize, not FD_ALLOC_BLOCK, in get_poll_index().
      gssd_proc: remove pointless test against FD_ALLOC_BLOCK in process_pipedir
      gssd: base the size of the fd array on the RLIMIT_NOFILE limit.


 utils/gssd/gssd_proc.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

-- 
Signature


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH 2/3] gssd_proc: remove pointless test against FD_ALLOC_BLOCK in process_pipedir
  2012-11-28  1:11 [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Neil Brown
  2012-11-28  1:11 ` [PATCH 1/3] gssd_proc: use pollsize, not FD_ALLOC_BLOCK, in get_poll_index() Neil Brown
@ 2012-11-28  1:11 ` Neil Brown
  2012-11-28  1:11 ` [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit Neil Brown
  2012-11-28 19:54 ` [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Steve Dickson
  3 siblings, 0 replies; 11+ messages in thread
From: Neil Brown @ 2012-11-28  1:11 UTC (permalink / raw)
  To: Steve Dickson; +Cc: linux-nfs

I can see no possible point for this test against FD_ALLOC_BLOCK,
so just remove the test.

Signed-off-by: NeilBrown <neilb@suse.de>
---
 utils/gssd/gssd_proc.c |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/utils/gssd/gssd_proc.c b/utils/gssd/gssd_proc.c
index 2d7ec94..425b582 100644
--- a/utils/gssd/gssd_proc.c
+++ b/utils/gssd/gssd_proc.c
@@ -567,9 +567,8 @@ process_pipedir(char *pipe_name)
 
 	update_old_clients(namelist, j, pipe_name);
 	for (i=0; i < j; i++) {
-		if (i < FD_ALLOC_BLOCK
-				&& !strncmp(namelist[i]->d_name, "clnt", 4)
-				&& !find_client(namelist[i]->d_name, pipe_name))
+		if (!strncmp(namelist[i]->d_name, "clnt", 4)
+		    && !find_client(namelist[i]->d_name, pipe_name))
 			process_clnt_dir(namelist[i]->d_name, pipe_name);
 		free(namelist[i]);
 	}



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
  2012-11-28  1:11 [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Neil Brown
  2012-11-28  1:11 ` [PATCH 1/3] gssd_proc: use pollsize, not FD_ALLOC_BLOCK, in get_poll_index() Neil Brown
  2012-11-28  1:11 ` [PATCH 2/3] gssd_proc: remove pointless test against FD_ALLOC_BLOCK in process_pipedir Neil Brown
@ 2012-11-28  1:11 ` Neil Brown
  2012-11-28 13:10   ` J. Bruce Fields
  2012-11-28 19:54 ` [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Steve Dickson
  3 siblings, 1 reply; 11+ messages in thread
From: Neil Brown @ 2012-11-28  1:11 UTC (permalink / raw)
  To: Steve Dickson; +Cc: linux-nfs

We have previously raised the size of the 'pollarray' once (32 -> 256)
and I have had another request to make it bigger.
Rather than changing the hard-coded value, make it depend on
RLIMIT_NOFILE.  This is an upper limit on the size of the array
that can be passed to poll() anyway.

Signed-off-by: NeilBrown <neilb@suse.de>
---
 utils/gssd/gssd_proc.c |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/utils/gssd/gssd_proc.c b/utils/gssd/gssd_proc.c
index 425b582..e32b2f0 100644
--- a/utils/gssd/gssd_proc.c
+++ b/utils/gssd/gssd_proc.c
@@ -52,6 +52,7 @@
 #include <sys/socket.h>
 #include <arpa/inet.h>
 #include <sys/fsuid.h>
+#include <sys/resource.h>
 
 #include <stdio.h>
 #include <stdlib.h>
@@ -483,9 +484,13 @@ fail_keep_client:
 void
 init_client_list(void)
 {
+	struct rlimit rlim;
 	TAILQ_INIT(&clnt_list);
 	/* Eventually plan to grow/shrink poll array: */
 	pollsize = FD_ALLOC_BLOCK;
+	if (getrlimit(RLIMIT_NOFILE, &rlim) < 0 &&
+	    rlim.rlim_cur != RLIM_INFINITY)
+		pollsize = rlim.rlim_cur;
 	pollarray = calloc(pollsize, sizeof(struct pollfd));
 }
 



^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
  2012-11-28  1:11 ` [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit Neil Brown
@ 2012-11-28 13:10   ` J. Bruce Fields
  2012-11-29  0:30     ` NeilBrown
  0 siblings, 1 reply; 11+ messages in thread
From: J. Bruce Fields @ 2012-11-28 13:10 UTC (permalink / raw)
  To: Neil Brown; +Cc: Steve Dickson, linux-nfs

On Wed, Nov 28, 2012 at 12:11:23PM +1100, Neil Brown wrote:
> We have previously raised the size of the 'pollarray' once (32 -> 256)
> and I have had another request to make it bigger.
> Rather than changing the hard-coded value, make it depend on
> RLIMIT_NOFILE.  This is an upper limit on the size of the array
> that can be passed to poll() anyway.

Sounds like a good idea.

Just out of curiosity: how does it fail?  I guess mounts just start
failing at some point--how do people find the workaround?

--b.

> 
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
>  utils/gssd/gssd_proc.c |    5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/utils/gssd/gssd_proc.c b/utils/gssd/gssd_proc.c
> index 425b582..e32b2f0 100644
> --- a/utils/gssd/gssd_proc.c
> +++ b/utils/gssd/gssd_proc.c
> @@ -52,6 +52,7 @@
>  #include <sys/socket.h>
>  #include <arpa/inet.h>
>  #include <sys/fsuid.h>
> +#include <sys/resource.h>
>  
>  #include <stdio.h>
>  #include <stdlib.h>
> @@ -483,9 +484,13 @@ fail_keep_client:
>  void
>  init_client_list(void)
>  {
> +	struct rlimit rlim;
>  	TAILQ_INIT(&clnt_list);
>  	/* Eventually plan to grow/shrink poll array: */
>  	pollsize = FD_ALLOC_BLOCK;
> +	if (getrlimit(RLIMIT_NOFILE, &rlim) < 0 &&
> +	    rlim.rlim_cur != RLIM_INFINITY)
> +		pollsize = rlim.rlim_cur;
>  	pollarray = calloc(pollsize, sizeof(struct pollfd));
>  }
>  
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic
  2012-11-28  1:11 [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Neil Brown
                   ` (2 preceding siblings ...)
  2012-11-28  1:11 ` [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit Neil Brown
@ 2012-11-28 19:54 ` Steve Dickson
  3 siblings, 0 replies; 11+ messages in thread
From: Steve Dickson @ 2012-11-28 19:54 UTC (permalink / raw)
  To: Neil Brown; +Cc: linux-nfs



On 27/11/12 20:11, Neil Brown wrote:
> We recently had a request to raise the number of fds that gssd can
> work with - again.
> 
> Rather than just change the hard-coded number I seemed to make sense
> to have it depend on RLIMIT_NOFILE - as poll won't work with more than
> that anyway.
> So the default becomes 1024, but it is easily raised further without
> code change.
> 
> These patches implement that.
All three packages were committed...

steved.

> 
> NeilBrown
> 
> 
> ---
> 
> Neil Brown (3):
>       gssd_proc: use pollsize, not FD_ALLOC_BLOCK, in get_poll_index().
>       gssd_proc: remove pointless test against FD_ALLOC_BLOCK in process_pipedir
>       gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
> 
> 
>  utils/gssd/gssd_proc.c |   14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
  2012-11-28 13:10   ` J. Bruce Fields
@ 2012-11-29  0:30     ` NeilBrown
  2012-12-11  0:02       ` NeilBrown
  0 siblings, 1 reply; 11+ messages in thread
From: NeilBrown @ 2012-11-29  0:30 UTC (permalink / raw)
  To: J. Bruce Fields; +Cc: Steve Dickson, linux-nfs

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

On Wed, 28 Nov 2012 08:10:55 -0500 "J. Bruce Fields" <bfields@fieldses.org>
wrote:

> On Wed, Nov 28, 2012 at 12:11:23PM +1100, Neil Brown wrote:
> > We have previously raised the size of the 'pollarray' once (32 -> 256)
> > and I have had another request to make it bigger.
> > Rather than changing the hard-coded value, make it depend on
> > RLIMIT_NOFILE.  This is an upper limit on the size of the array
> > that can be passed to poll() anyway.
> 
> Sounds like a good idea.
> 
> Just out of curiosity: how does it fail?  I guess mounts just start
> failing at some point--how do people find the workaround?

Error seems to be

rpcsec_gss: gss_init_sec_context: (major) Miscellaneous failure - (minor) Cannot contact any KDC for requested realm

in rpc.gssd logs.

I guess people could read the source to find the work around .... not ideal
though.  I guess we should get gssd to generate some more helpful message.

The seem to be further problems that the customer is experiencing so I might
wait until they are completely resolved to ensure I have complete
understanding before I propose a further patch.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
  2012-11-29  0:30     ` NeilBrown
@ 2012-12-11  0:02       ` NeilBrown
  2012-12-11 16:16         ` J. Bruce Fields
  0 siblings, 1 reply; 11+ messages in thread
From: NeilBrown @ 2012-12-11  0:02 UTC (permalink / raw)
  To: J. Bruce Fields; +Cc: Steve Dickson, linux-nfs

[-- Attachment #1: Type: text/plain, Size: 3375 bytes --]

On Thu, 29 Nov 2012 11:30:51 +1100 NeilBrown <neilb@suse.de> wrote:

> On Wed, 28 Nov 2012 08:10:55 -0500 "J. Bruce Fields" <bfields@fieldses.org>
> wrote:
> 
> > On Wed, Nov 28, 2012 at 12:11:23PM +1100, Neil Brown wrote:
> > > We have previously raised the size of the 'pollarray' once (32 -> 256)
> > > and I have had another request to make it bigger.
> > > Rather than changing the hard-coded value, make it depend on
> > > RLIMIT_NOFILE.  This is an upper limit on the size of the array
> > > that can be passed to poll() anyway.
> > 
> > Sounds like a good idea.
> > 
> > Just out of curiosity: how does it fail?  I guess mounts just start
> > failing at some point--how do people find the workaround?
> 
> Error seems to be
> 
> rpcsec_gss: gss_init_sec_context: (major) Miscellaneous failure - (minor) Cannot contact any KDC for requested realm
> 
> in rpc.gssd logs.
> 
> I guess people could read the source to find the work around .... not ideal
> though.  I guess we should get gssd to generate some more helpful message.
> 
> The seem to be further problems that the customer is experiencing so I might
> wait until they are completely resolved to ensure I have complete
> understanding before I propose a further patch.

The "further problem" was that krb5 libraries use select() in a way that does
not support file descriptors higher than 1024.  This is fixed in the latest
krb5 so that is no longer an issue.

I've been thinking about your question, and about how best to deliver a fix
to customers, and I really think it should all "just work".
i.e. the array that gssd should be sized dynamically and RLIMIT_NOFILE should
be increased as needed.

I haven't tested this, but what do people think?   I don't have a problem
changing the rlim_cur limit like this, but I wonder if it is OK to
dynamically set rlim_max.

Thoughts?

NeilBrown


diff --git a/utils/gssd/gssd_proc.c b/utils/gssd/gssd_proc.c
index d01ba2f..3576a6f 100644
--- a/utils/gssd/gssd_proc.c
+++ b/utils/gssd/gssd_proc.c
@@ -389,18 +389,36 @@ static int
 get_poll_index(int *ind)
 {
 	unsigned int i;
+	struct pollfd *new_pollarray;
+	struct rlimit rlim;
 
 	*ind = -1;
 	for (i=0; i<pollsize; i++) {
 		if (pollarray[i].events == 0) {
 			*ind = i;
-			break;
+			return 0;
 		}
 	}
-	if (*ind == -1) {
+
+	new_pollarray = realloc(pollarray, pollsize * 2 * sizeof(*pollarray));
+	if (!new_pollarray) {
 		printerr(0, "ERROR: No pollarray slots open\n");
 		return -1;
 	}
+	pollarray = new_pollarray;
+	memset(pollarray + pollsize, 0, sizeof(*pollarray) * pollsize);
+	*ind = pollsize;
+	pollsize *= 2;
+
+	/* We will need lots of file descriptors too */
+	if (getrlimit(RLIMIT_NOFILE, &rlim) == 0) {
+		if (rlim.rlim_cur < pollsize+20) {
+			rlim.rlim_cur = pollsize + 20;
+			if (rlim.rlim_max < rlim.rlim_cur)
+				rlim.rlim_max = rlim.rlim_cur;
+			setrlimit(RLIMIT_NOFILE, &rlim);
+		}
+	}
 	return 0;
 }
 
@@ -473,13 +491,9 @@ fail_keep_client:
 void
 init_client_list(void)
 {
-	struct rlimit rlim;
 	TAILQ_INIT(&clnt_list);
 	/* Eventually plan to grow/shrink poll array: */
 	pollsize = FD_ALLOC_BLOCK;
-	if (getrlimit(RLIMIT_NOFILE, &rlim) < 0 &&
-	    rlim.rlim_cur != RLIM_INFINITY)
-		pollsize = rlim.rlim_cur;
 	pollarray = calloc(pollsize, sizeof(struct pollfd));
 }
 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
  2012-12-11  0:02       ` NeilBrown
@ 2012-12-11 16:16         ` J. Bruce Fields
  2012-12-13  6:03           ` NeilBrown
  0 siblings, 1 reply; 11+ messages in thread
From: J. Bruce Fields @ 2012-12-11 16:16 UTC (permalink / raw)
  To: NeilBrown; +Cc: Steve Dickson, linux-nfs

On Tue, Dec 11, 2012 at 11:02:28AM +1100, NeilBrown wrote:
> On Thu, 29 Nov 2012 11:30:51 +1100 NeilBrown <neilb@suse.de> wrote:
> 
> > On Wed, 28 Nov 2012 08:10:55 -0500 "J. Bruce Fields" <bfields@fieldses.org>
> > wrote:
> > 
> > > On Wed, Nov 28, 2012 at 12:11:23PM +1100, Neil Brown wrote:
> > > > We have previously raised the size of the 'pollarray' once (32 -> 256)
> > > > and I have had another request to make it bigger.
> > > > Rather than changing the hard-coded value, make it depend on
> > > > RLIMIT_NOFILE.  This is an upper limit on the size of the array
> > > > that can be passed to poll() anyway.
> > > 
> > > Sounds like a good idea.
> > > 
> > > Just out of curiosity: how does it fail?  I guess mounts just start
> > > failing at some point--how do people find the workaround?
> > 
> > Error seems to be
> > 
> > rpcsec_gss: gss_init_sec_context: (major) Miscellaneous failure - (minor) Cannot contact any KDC for requested realm
> > 
> > in rpc.gssd logs.
> > 
> > I guess people could read the source to find the work around .... not ideal
> > though.  I guess we should get gssd to generate some more helpful message.
> > 
> > The seem to be further problems that the customer is experiencing so I might
> > wait until they are completely resolved to ensure I have complete
> > understanding before I propose a further patch.
> 
> The "further problem" was that krb5 libraries use select() in a way that does
> not support file descriptors higher than 1024.  This is fixed in the latest
> krb5 so that is no longer an issue.
> 
> I've been thinking about your question, and about how best to deliver a fix
> to customers, and I really think it should all "just work".
> i.e. the array that gssd should be sized dynamically and RLIMIT_NOFILE should
> be increased as needed.

Neat-o.

> I haven't tested this, but what do people think?   I don't have a problem
> changing the rlim_cur limit like this, but I wonder if it is OK to
> dynamically set rlim_max.

What would be the concern, that we'd be depriving an admin of the
ability to set a limit?

We could instead set only the current limit and set set the max to an
admin-configurable quantity (default very large) when we start gssd.

But that sounds more complicated, and off hand I can't think of a reason
an admin would want to do that.

--b.

> 
> Thoughts?
> 
> NeilBrown
> 
> 
> diff --git a/utils/gssd/gssd_proc.c b/utils/gssd/gssd_proc.c
> index d01ba2f..3576a6f 100644
> --- a/utils/gssd/gssd_proc.c
> +++ b/utils/gssd/gssd_proc.c
> @@ -389,18 +389,36 @@ static int
>  get_poll_index(int *ind)
>  {
>  	unsigned int i;
> +	struct pollfd *new_pollarray;
> +	struct rlimit rlim;
>  
>  	*ind = -1;
>  	for (i=0; i<pollsize; i++) {
>  		if (pollarray[i].events == 0) {
>  			*ind = i;
> -			break;
> +			return 0;
>  		}
>  	}
> -	if (*ind == -1) {
> +
> +	new_pollarray = realloc(pollarray, pollsize * 2 * sizeof(*pollarray));
> +	if (!new_pollarray) {
>  		printerr(0, "ERROR: No pollarray slots open\n");
>  		return -1;
>  	}
> +	pollarray = new_pollarray;
> +	memset(pollarray + pollsize, 0, sizeof(*pollarray) * pollsize);
> +	*ind = pollsize;
> +	pollsize *= 2;
> +
> +	/* We will need lots of file descriptors too */
> +	if (getrlimit(RLIMIT_NOFILE, &rlim) == 0) {
> +		if (rlim.rlim_cur < pollsize+20) {
> +			rlim.rlim_cur = pollsize + 20;
> +			if (rlim.rlim_max < rlim.rlim_cur)
> +				rlim.rlim_max = rlim.rlim_cur;
> +			setrlimit(RLIMIT_NOFILE, &rlim);
> +		}
> +	}
>  	return 0;
>  }
>  
> @@ -473,13 +491,9 @@ fail_keep_client:
>  void
>  init_client_list(void)
>  {
> -	struct rlimit rlim;
>  	TAILQ_INIT(&clnt_list);
>  	/* Eventually plan to grow/shrink poll array: */
>  	pollsize = FD_ALLOC_BLOCK;
> -	if (getrlimit(RLIMIT_NOFILE, &rlim) < 0 &&
> -	    rlim.rlim_cur != RLIM_INFINITY)
> -		pollsize = rlim.rlim_cur;
>  	pollarray = calloc(pollsize, sizeof(struct pollfd));
>  }
>  



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
  2012-12-11 16:16         ` J. Bruce Fields
@ 2012-12-13  6:03           ` NeilBrown
  2012-12-13 19:21             ` J. Bruce Fields
  0 siblings, 1 reply; 11+ messages in thread
From: NeilBrown @ 2012-12-13  6:03 UTC (permalink / raw)
  To: J. Bruce Fields; +Cc: Steve Dickson, linux-nfs

[-- Attachment #1: Type: text/plain, Size: 3134 bytes --]

On Tue, 11 Dec 2012 11:16:33 -0500 "J. Bruce Fields" <bfields@fieldses.org>
wrote:

> On Tue, Dec 11, 2012 at 11:02:28AM +1100, NeilBrown wrote:
> > On Thu, 29 Nov 2012 11:30:51 +1100 NeilBrown <neilb@suse.de> wrote:
> > 
> > > On Wed, 28 Nov 2012 08:10:55 -0500 "J. Bruce Fields" <bfields@fieldses.org>
> > > wrote:
> > > 
> > > > On Wed, Nov 28, 2012 at 12:11:23PM +1100, Neil Brown wrote:
> > > > > We have previously raised the size of the 'pollarray' once (32 -> 256)
> > > > > and I have had another request to make it bigger.
> > > > > Rather than changing the hard-coded value, make it depend on
> > > > > RLIMIT_NOFILE.  This is an upper limit on the size of the array
> > > > > that can be passed to poll() anyway.
> > > > 
> > > > Sounds like a good idea.
> > > > 
> > > > Just out of curiosity: how does it fail?  I guess mounts just start
> > > > failing at some point--how do people find the workaround?
> > > 
> > > Error seems to be
> > > 
> > > rpcsec_gss: gss_init_sec_context: (major) Miscellaneous failure - (minor) Cannot contact any KDC for requested realm
> > > 
> > > in rpc.gssd logs.
> > > 
> > > I guess people could read the source to find the work around .... not ideal
> > > though.  I guess we should get gssd to generate some more helpful message.
> > > 
> > > The seem to be further problems that the customer is experiencing so I might
> > > wait until they are completely resolved to ensure I have complete
> > > understanding before I propose a further patch.
> > 
> > The "further problem" was that krb5 libraries use select() in a way that does
> > not support file descriptors higher than 1024.  This is fixed in the latest
> > krb5 so that is no longer an issue.
> > 
> > I've been thinking about your question, and about how best to deliver a fix
> > to customers, and I really think it should all "just work".
> > i.e. the array that gssd should be sized dynamically and RLIMIT_NOFILE should
> > be increased as needed.
> 
> Neat-o.
> 
> > I haven't tested this, but what do people think?   I don't have a problem
> > changing the rlim_cur limit like this, but I wonder if it is OK to
> > dynamically set rlim_max.
> 
> What would be the concern, that we'd be depriving an admin of the
> ability to set a limit?

My concern in that automagically raising a so-called "hard limit" seems to be
subverting the very concept of it being a "limit".

> 
> We could instead set only the current limit and set set the max to an
> admin-configurable quantity (default very large) when we start gssd.

I really want to avoid any configuration.
The number of fds that will be used is directly connected to the number of
concurrent mounts - as there is no limit on those (I assume), I guess it is
fair that there is no limit on fds used by gssd.

> 
> But that sounds more complicated, and off hand I can't think of a reason
> an admin would want to do that.

OK, let's just modify the hard limit dynamically ... though I'm about to
disappear for summer holidays so I doubt you'll see anything for some weeks.


thanks,
NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit.
  2012-12-13  6:03           ` NeilBrown
@ 2012-12-13 19:21             ` J. Bruce Fields
  0 siblings, 0 replies; 11+ messages in thread
From: J. Bruce Fields @ 2012-12-13 19:21 UTC (permalink / raw)
  To: NeilBrown; +Cc: Steve Dickson, linux-nfs

On Thu, Dec 13, 2012 at 05:03:22PM +1100, NeilBrown wrote:
> On Tue, 11 Dec 2012 11:16:33 -0500 "J. Bruce Fields" <bfields@fieldses.org>
> wrote:
> 
> > On Tue, Dec 11, 2012 at 11:02:28AM +1100, NeilBrown wrote:
> > > On Thu, 29 Nov 2012 11:30:51 +1100 NeilBrown <neilb@suse.de> wrote:
> > > 
> > > > On Wed, 28 Nov 2012 08:10:55 -0500 "J. Bruce Fields" <bfields@fieldses.org>
> > > > wrote:
> > > > 
> > > > > On Wed, Nov 28, 2012 at 12:11:23PM +1100, Neil Brown wrote:
> > > > > > We have previously raised the size of the 'pollarray' once (32 -> 256)
> > > > > > and I have had another request to make it bigger.
> > > > > > Rather than changing the hard-coded value, make it depend on
> > > > > > RLIMIT_NOFILE.  This is an upper limit on the size of the array
> > > > > > that can be passed to poll() anyway.
> > > > > 
> > > > > Sounds like a good idea.
> > > > > 
> > > > > Just out of curiosity: how does it fail?  I guess mounts just start
> > > > > failing at some point--how do people find the workaround?
> > > > 
> > > > Error seems to be
> > > > 
> > > > rpcsec_gss: gss_init_sec_context: (major) Miscellaneous failure - (minor) Cannot contact any KDC for requested realm
> > > > 
> > > > in rpc.gssd logs.
> > > > 
> > > > I guess people could read the source to find the work around .... not ideal
> > > > though.  I guess we should get gssd to generate some more helpful message.
> > > > 
> > > > The seem to be further problems that the customer is experiencing so I might
> > > > wait until they are completely resolved to ensure I have complete
> > > > understanding before I propose a further patch.
> > > 
> > > The "further problem" was that krb5 libraries use select() in a way that does
> > > not support file descriptors higher than 1024.  This is fixed in the latest
> > > krb5 so that is no longer an issue.
> > > 
> > > I've been thinking about your question, and about how best to deliver a fix
> > > to customers, and I really think it should all "just work".
> > > i.e. the array that gssd should be sized dynamically and RLIMIT_NOFILE should
> > > be increased as needed.
> > 
> > Neat-o.
> > 
> > > I haven't tested this, but what do people think?   I don't have a problem
> > > changing the rlim_cur limit like this, but I wonder if it is OK to
> > > dynamically set rlim_max.
> > 
> > What would be the concern, that we'd be depriving an admin of the
> > ability to set a limit?
> 
> My concern in that automagically raising a so-called "hard limit" seems to be
> subverting the very concept of it being a "limit".
> 
> > 
> > We could instead set only the current limit and set set the max to an
> > admin-configurable quantity (default very large) when we start gssd.
> 
> I really want to avoid any configuration.

Well, the init scripts (or whatever we use these days) would need to be
modified to set the max to RLIMIT_INFINITY by default, but the admin
shouldn't ever have to do anything.

But honestly I can't see any practical advantage to that, so...

> The number of fds that will be used is directly connected to the number of
> concurrent mounts - as there is no limit on those (I assume), I guess it is
> fair that there is no limit on fds used by gssd.
> 
> > 
> > But that sounds more complicated, and off hand I can't think of a reason
> > an admin would want to do that.
> 
> OK, let's just modify the hard limit dynamically ...

... fine by me.

> though I'm about to
> disappear for summer holidays so I doubt you'll see anything for some weeks.

"Summer holidays", huh.

Enjoy!

--b.

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2012-12-13 19:21 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-11-28  1:11 [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Neil Brown
2012-11-28  1:11 ` [PATCH 1/3] gssd_proc: use pollsize, not FD_ALLOC_BLOCK, in get_poll_index() Neil Brown
2012-11-28  1:11 ` [PATCH 2/3] gssd_proc: remove pointless test against FD_ALLOC_BLOCK in process_pipedir Neil Brown
2012-11-28  1:11 ` [PATCH 3/3] gssd: base the size of the fd array on the RLIMIT_NOFILE limit Neil Brown
2012-11-28 13:10   ` J. Bruce Fields
2012-11-29  0:30     ` NeilBrown
2012-12-11  0:02       ` NeilBrown
2012-12-11 16:16         ` J. Bruce Fields
2012-12-13  6:03           ` NeilBrown
2012-12-13 19:21             ` J. Bruce Fields
2012-11-28 19:54 ` [PATCH 0/3] Make size of gssd_proc fd array a little more dynamic Steve Dickson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.