Wrong network

Wrong network

[Steve Thompson]

CentOS 5.5 (32- and 64-bit), autofs-5.0.1-0.rc2.143.el5_5.6.

Two subnets, A (192.168.0/22) and B (192.168.4/22). NFS server has an 
interface on both A and B. NFS client has an interface on B only. Maps 
come via LDAP. On the client:

 	# mount
 	<server>:/mnt/point on /fs/<blah> type nfs (rw,addr=192.168.0.69)

The addr= looks wrong to me; should be 192.168.4.x. On other clients, the 
subnet pointed to by addr= is sometimes right, sometimes wrong. What's 
wrong here?

-Steve

Re: Wrong network

[Frank Thommen]

On 12.02.2011 15:29, Steve Thompson wrote:
> CentOS 5.5 (32- and 64-bit), autofs-5.0.1-0.rc2.143.el5_5.6.
>
> Two subnets, A (192.168.0/22) and B (192.168.4/22). NFS server has an
> interface on both A and B. NFS client has an interface on B only. Maps
> come via LDAP. On the client:
>
> # mount
> <server>:/mnt/point on /fs/<blah> type nfs (rw,addr=192.168.0.69)
>
> The addr= looks wrong to me; should be 192.168.4.x. On other clients,
> the subnet pointed to by addr= is sometimes right, sometimes wrong.
> What's wrong here?


Wrong network/gateway configuration on the NFS server (or on the clients)?

frank

Re: Wrong network

[Steve Thompson]

On Sat, 12 Feb 2011, Frank Thommen wrote:

> On 12.02.2011 15:29, Steve Thompson wrote:
>> CentOS 5.5 (32- and 64-bit), autofs-5.0.1-0.rc2.143.el5_5.6.
>> 
>> Two subnets, A (192.168.0/22) and B (192.168.4/22). NFS server has an
>> interface on both A and B. NFS client has an interface on B only. Maps
>> come via LDAP. On the client:
>> 
>> # mount
>> <server>:/mnt/point on /fs/<blah> type nfs (rw,addr=192.168.0.69)
>> 
>> The addr= looks wrong to me; should be 192.168.4.x. On other clients,
>> the subnet pointed to by addr= is sometimes right, sometimes wrong.
>> What's wrong here?
>
> Wrong network/gateway configuration on the NFS server (or on the clients)?

Thanks for the input, but the setup on the NFS server and clients is 
correct in every aspect as far as I can tell.

-s

Re: Wrong network

[Steve Thompson]

On Sat, 12 Feb 2011, Steve Thompson wrote:

> CentOS 5.5 (32- and 64-bit), autofs-5.0.1-0.rc2.143.el5_5.6.
>
> Two subnets, A (192.168.0/22) and B (192.168.4/22). NFS server has an 
> interface on both A and B. NFS client has an interface on B only. Maps come 
> via LDAP. On the client:
>
> 	# mount
> 	<server>:/mnt/point on /fs/<blah> type nfs (rw,addr=192.168.0.69)
>
> The addr= looks wrong to me; should be 192.168.4.x. On other clients, the 
> subnet pointed to by addr= is sometimes right, sometimes wrong. What's wrong 
> here?

Let me expand a little on this. A DNS lookup of the NFS server name 
returns two IP addresses, one in subnet A and one in subnet B. I was under 
the impression that autofs would see that the address on subnet B is in 
the same segment as the client, using PROXIMITY_SUBNET. It doesn't appear 
to be working the way I would expect.

Steve

Re: Wrong network

[Ian Kent]

On Sat, 2011-02-12 at 15:21 -0500, Steve Thompson wrote:
> On Sat, 12 Feb 2011, Steve Thompson wrote:
> 
> > CentOS 5.5 (32- and 64-bit), autofs-5.0.1-0.rc2.143.el5_5.6.
> >
> > Two subnets, A (192.168.0/22) and B (192.168.4/22). NFS server has an 
> > interface on both A and B. NFS client has an interface on B only. Maps come 
> > via LDAP. On the client:
> >
> > 	# mount
> > 	<server>:/mnt/point on /fs/<blah> type nfs (rw,addr=192.168.0.69)
> >
> > The addr= looks wrong to me; should be 192.168.4.x. On other clients, the 
> > subnet pointed to by addr= is sometimes right, sometimes wrong. What's wrong 
> > here?
> 
> Let me expand a little on this. A DNS lookup of the NFS server name 
> returns two IP addresses, one in subnet A and one in subnet B. I was under 
> the impression that autofs would see that the address on subnet B is in 
> the same segment as the client, using PROXIMITY_SUBNET. It doesn't appear 
> to be working the way I would expect.

That is the way it should work but dns round robin entries like this
aren't handled quite correctly.

I'm not sure this patch will apply (the CHANGELOG hunk certainly won't)
but you could try it:

autofs-5.0.5 - mount using address for rr

From: Ian Kent <raven@themaw.net>

When a host has multiple addresses, mount using individual address so
we can take advantage of the probing and response time calculation that's
already been done.
---

 CHANGELOG            |    1 +
 include/replicated.h |    1 +
 modules/mount_nfs.c  |   31 +++++++++++++++++++++++++------
 modules/replicated.c |   11 ++++++++---
 4 files changed, 35 insertions(+), 9 deletions(-)


--- autofs-5.0.5.orig/CHANGELOG
+++ autofs-5.0.5/CHANGELOG
@@ -42,6 +42,7 @@
 - use weight only for server selection.
 - fix isspace() wild card substition.
 - include ip address in debug logging.
+- mount using address for DNS round robin host names.
 
 03/09/2009 autofs-5.0.5
 -----------------------
--- autofs-5.0.5.orig/include/replicated.h
+++ autofs-5.0.5/include/replicated.h
@@ -54,6 +54,7 @@ struct host {
 	char *name;
 	struct sockaddr *addr;
 	size_t addr_len;
+	unsigned int rr;
 	char *path;
 	unsigned int version;
 	unsigned int options;
--- autofs-5.0.5.orig/modules/mount_nfs.c
+++ autofs-5.0.5/modules/mount_nfs.c
@@ -229,13 +229,32 @@ int mount_mount(struct autofs_point *ap,
 
 		/* Not a local host - do an NFS mount */
 
-		loc = malloc(strlen(this->name) + 1 + strlen(this->path) + 1);
-		if (!loc) {
-			char *estr = strerror_r(errno, buf, MAX_ERR_BUF);
-			error(ap->logopt, "malloc: %s", estr);
-			return 1;
+		if (this->rr && this->addr) {
+			socklen_t len = INET6_ADDRSTRLEN;
+			char buf[len + 1];
+			const char *n_addr;
+			n_addr = get_addr_string(this->addr, buf, len);
+			loc = malloc(strlen(n_addr) + strlen(this->path) + 4);
+			if (!loc) {
+				char *estr = strerror_r(errno, buf, MAX_ERR_BUF);
+				error(ap->logopt, "malloc: %s", estr);
+				return 1;
+			}
+			if (this->addr->sa_family == AF_INET6) {
+				strcpy(loc, "[");
+				strcat(loc, n_addr);
+				strcat(loc, "]");
+			} else
+				strcpy(loc, n_addr);
+		} else {
+			loc = malloc(strlen(this->name) + strlen(this->path) + 2);
+			if (!loc) {
+				char *estr = strerror_r(errno, buf, MAX_ERR_BUF);
+				error(ap->logopt, "malloc: %s", estr);
+				return 1;
+			}
+			strcpy(loc, this->name);
 		}
-		strcpy(loc, this->name);
 		strcat(loc, ":");
 		strcat(loc, this->path);
 
--- autofs-5.0.5.orig/modules/replicated.c
+++ autofs-5.0.5/modules/replicated.c
@@ -1059,7 +1059,8 @@ int prune_host_list(unsigned logopt, str
 
 static int add_new_host(struct host **list,
 			const char *host, unsigned int weight,
-			struct addrinfo *host_addr, unsigned int options)
+			struct addrinfo *host_addr,
+			unsigned int rr, unsigned int options)
 {
 	struct host *new;
 	unsigned int prx;
@@ -1094,6 +1095,7 @@ static int add_new_host(struct host **li
 		free_host(new);
 		return 0;
 	}
+	new->rr = rr;
 
 	return 1;
 }
@@ -1102,6 +1104,7 @@ static int add_host_addrs(struct host **
 			  unsigned int weight, unsigned int options)
 {
 	struct addrinfo hints, *ni, *this;
+	int rr = 0;
 	int ret;
 
 	memset(&hints, 0, sizeof(hints));
@@ -1115,7 +1118,7 @@ static int add_host_addrs(struct host **
 
 	this = ni;
 	while (this) {
-		ret = add_new_host(list, host, weight, this, options);
+		ret = add_new_host(list, host, weight, this, 0, options);
 		if (!ret)
 			break;
 		this = this->ai_next;
@@ -1137,8 +1140,10 @@ try_name:
 	}
 
 	this = ni;
+	if (this->ai_next)
+		rr++;
 	while (this) {
-		ret = add_new_host(list, host, weight, this, options);
+		ret = add_new_host(list, host, weight, this, rr, options);
 		if (!ret)
 			break;
 		this = this->ai_next;

Re: Wrong network

[Steve Thompson]

On Mon, 14 Feb 2011, Ian Kent wrote:

> That is the way it should work but dns round robin entries like this
> aren't handled quite correctly.
>
> I'm not sure this patch will apply (the CHANGELOG hunk certainly won't)
> but you could try it:

Unfortunately I am not able to try this - I can use only the .rpm's that 
come from the CentOS repo. Hopefully this patch will make it upstream 
soon, given the seriousness of this issue.

Steve

Re: Wrong network

[Ian Kent]

On Mon, 2011-02-14 at 14:22 -0500, Steve Thompson wrote:
> On Mon, 14 Feb 2011, Ian Kent wrote:
> 
> > That is the way it should work but dns round robin entries like this
> > aren't handled quite correctly.
> >
> > I'm not sure this patch will apply (the CHANGELOG hunk certainly won't)
> > but you could try it:
> 
> Unfortunately I am not able to try this - I can use only the .rpm's that 
> come from the CentOS repo. Hopefully this patch will make it upstream 
> soon, given the seriousness of this issue.

That's a lame response, given you think it's a serious issue.
I guess you won't be prepared to test a pre-built package either.
If you are willing to test the change try:
http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.145.bz520844.2.el5

Ian

Re: Wrong network

[Steve Thompson]

On Tue, 15 Feb 2011, Ian Kent wrote:

> That's a lame response, given you think it's a serious issue.

I don't think that is a fair response either. For all you know, I might be 
limited by site policy.

Re: Wrong network

[Ian Kent]

On Tue, 2011-02-15 at 07:04 -0500, Steve Thompson wrote:
> On Tue, 15 Feb 2011, Ian Kent wrote:
> 
> > That's a lame response, given you think it's a serious issue.
> 
> I don't think that is a fair response either. For all you know, I might be 
> limited by site policy.

Fair call, but it comes over as though you don't want to contribute at
all, which isn't good. If you post to the upstream list I more or less
expect you will are willing to reserve a couple of machines for testing,
that you have a fairly good level of know how and that you will put a
bit of time into testing corrections. Of course some bugs are hard to
duplicate for one reason or another and that's a bit different.

Ian

Re: Wrong network

[Steve Thompson]

On Tue, 15 Feb 2011, Ian Kent wrote:

> Fair call, but it comes over as though you don't want to contribute at
> all, which isn't good.

That's not the impression I meant to give at all. I _am_ building some new 
test machines (about ten of them), and I _will_ do some testing. My test 
machines will all be 32-bit, though, as there is no spare 64-bit hardware. 
I just can't go testing on a production network; what with this and the 
hang problem I described in a separate post, I'm close enough to being 
lynched as it is. I have written my own applications of a similar level of 
complexity to autofs, and I certainly appreciate that it's not easy, and I 
definitely appreciate the job that you're doing.

Steve

Re: Wrong network

[Ian Kent]

On Tue, 2011-02-15 at 07:40 -0500, Steve Thompson wrote:
> On Tue, 15 Feb 2011, Ian Kent wrote:
> 
> > Fair call, but it comes over as though you don't want to contribute at
> > all, which isn't good.
> 
> That's not the impression I meant to give at all. I _am_ building some new 
> test machines (about ten of them), and I _will_ do some testing. My test 
> machines will all be 32-bit, though, as there is no spare 64-bit hardware. 
> I just can't go testing on a production network; what with this and the 
> hang problem I described in a separate post, I'm close enough to being 
> lynched as it is. I have written my own applications of a similar level of 
> complexity to autofs, and I certainly appreciate that it's not easy, and I 
> definitely appreciate the job that you're doing.

We're good then.

I appreciate you are in a difficult spot, but spare a thought for the
pressures I may have that tent to make me a bit short from time to
time, ;)

As far as the hang you have seen, I don't know why that's happening, the
patches were added between el5_5.4 and el5_5.6 have been around for
quite a while, upstream and in Fedora and tested by more than one
customer, so I didn't expect to hear of a problem. Like I said, send me
a gdb backtrace so that I can see where it's happening.

Just to confirm, this is autofs-5.0.1-0.rc2.143.el5_5.6, and what kernel
revision, I guess the same as RHEL-5.5, which should be fine.

I can post the previous RedHat package to people.redhat.com so you can
revert back while we work on this, if that will kelp.

Ian

Re: Wrong network

[Steve Thompson]

On Tue, 15 Feb 2011, Ian Kent wrote:

> As far as the hang you have seen, I don't know why that's happening, the
> patches were added between el5_5.4 and el5_5.6 have been around for
> quite a while, upstream and in Fedora and tested by more than one
> customer, so I didn't expect to hear of a problem.

While thinking about this I noticed that the automount hangs all occurred 
on systems that had SElinux set to enforcing mode; there were no hangs on 
systems with SElinux either disabled or in permissive mode. So for now I 
have disabled SElinux on most of the systems on which a hang was observed. 
So far, so good: no more hangs. But it's only been a couple of days, so I 
will keep an eye on it.

Steve

Re: Wrong network

[Ian Kent]

On Wed, 2011-02-16 at 17:00 -0500, Steve Thompson wrote:
> On Tue, 15 Feb 2011, Ian Kent wrote:
> 
> > As far as the hang you have seen, I don't know why that's happening, the
> > patches were added between el5_5.4 and el5_5.6 have been around for
> > quite a while, upstream and in Fedora and tested by more than one
> > customer, so I didn't expect to hear of a problem.
> 
> While thinking about this I noticed that the automount hangs all occurred 
> on systems that had SElinux set to enforcing mode; there were no hangs on 
> systems with SElinux either disabled or in permissive mode. So for now I 
> have disabled SElinux on most of the systems on which a hang was observed. 
> So far, so good: no more hangs. But it's only been a couple of days, so I 
> will keep an eye on it.

Right, but to help me a gdb backtrace (the "thr a a bt") of a hung
automount process would be useful.

Also, an avc log from a system with selinux in permissive mode is needed
to find out what is happening. I don't think the changes to the package
caused this breakage and QA testing is always done with selinux in
enforcing mode. So maybe there has been an selinux policy change too.

Ian