Replicated hosts in autofs v4.

Replicated hosts in autofs v4.

[Jeremy Rosengren]

I've previously posted about this problem, but I still haven't found
an answer, although I think I better understand what's happening.

We have an automount map called "auto_group" that looks like this:

icdes     ausfiler,minnfiler:/vol/vol0/data/group/&

ausfiler lives in Texas, minnfiler lives in Minnesota.  Between
ausfiler and minnfiler lies a T1 line.

On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3
client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts
ausfiler:/vol/vol0/data/group/icdes.  No matter what...it always
mounts ausfiler.  The problem is that ausfiler is on the other end of
a T1 from the client, whereas minnfiler is on the same subnet, on the
same network switch.

I filed a support ticket with RedHat Global support, and an engineer
there confirmed that the client does an RPC call with a  .1 second
timeout.  However, the client's behavior indicates to me that the
comparison being made in the reponse times between the replicated
mount servers isn't fine-grained enough to make the correct choice.  A
coworker made the comment that our T1 line could be "too robust",
making the server farther away appear to be good enough for the
client's purposes.

Solaris doesn't seem to have any issues with doing this properly.  A
Solaris client in Minnesota will always mount the "closest" server in
a replicated host map, which is important because we're trying to use
RHEL3 to replace some of our Solaris infrastructure.  The RH Global
support engineer suggested reversing the order of the servers in the
map, however that would cause clients in Texas to exhibit the same
behavior that clients in Minneapolis are.

Can somebody explain to me whether I'm not the right track in
understanding the problem, with regards to the comparison not being
fine-grained enough for the client to make the correct choice?  Or,
might there be something else going on?  I previously had assumed that
the behavior I was seeing was related to the bug that caused the
client to always mount the first-listed server, but I no longer think
that's the case.

Thanks,

-- jeremy

Re: Replicated hosts in autofs v4.

[Denis Vlasenko]

On Sunday 10 October 2004 19:18, Jeremy Rosengren wrote:
> I've previously posted about this problem, but I still haven't found
> an answer, although I think I better understand what's happening.
> 
> We have an automount map called "auto_group" that looks like this:
> 
> icdes     ausfiler,minnfiler:/vol/vol0/data/group/&
> 
> ausfiler lives in Texas, minnfiler lives in Minnesota.  Between
> ausfiler and minnfiler lies a T1 line.
> 
> On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3
> client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts
> ausfiler:/vol/vol0/data/group/icdes.  No matter what...it always
> mounts ausfiler.  The problem is that ausfiler is on the other end of
> a T1 from the client, whereas minnfiler is on the same subnet, on the
> same network switch.

Your config does not let the system know that you prefer minnfiler
over ausfiler if both are available.

> I filed a support ticket with RedHat Global support, and an engineer
> there confirmed that the client does an RPC call with a  .1 second
> timeout.  However, the client's behavior indicates to me that the
> comparison being made in the reponse times between the replicated
> mount servers isn't fine-grained enough to make the correct choice.  A
> coworker made the comment that our T1 line could be "too robust",
> making the server farther away appear to be good enough for the
> client's purposes.
> 
> Solaris doesn't seem to have any issues with doing this properly.  A

Prolly Solaris does not use first server from the set of those which
answered, it uses server which _answers first_.
--
vda

Re: Replicated hosts in autofs v4.

[Ian Kent]

On Mon, 2004-10-11 at 00:18, Jeremy Rosengren wrote:
> I've previously posted about this problem, but I still haven't found
> an answer, although I think I better understand what's happening.
> 
> We have an automount map called "auto_group" that looks like this:
> 
> icdes     ausfiler,minnfiler:/vol/vol0/data/group/&
> 
> ausfiler lives in Texas, minnfiler lives in Minnesota.  Between
> ausfiler and minnfiler lies a T1 line.
> 
> On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3
> client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts
> ausfiler:/vol/vol0/data/group/icdes.  No matter what...it always
> mounts ausfiler.  The problem is that ausfiler is on the other end of
> a T1 from the client, whereas minnfiler is on the same subnet, on the
> same network switch.
> 
> I filed a support ticket with RedHat Global support, and an engineer
> there confirmed that the client does an RPC call with a  .1 second
> timeout.  However, the client's behavior indicates to me that the
> comparison being made in the reponse times between the replicated
> mount servers isn't fine-grained enough to make the correct choice.  A
> coworker made the comment that our T1 line could be "too robust",
> making the server farther away appear to be good enough for the
> client's purposes.
> 
> Solaris doesn't seem to have any issues with doing this properly.  A
> Solaris client in Minnesota will always mount the "closest" server in
> a replicated host map, which is important because we're trying to use
> RHEL3 to replace some of our Solaris infrastructure.  The RH Global
> support engineer suggested reversing the order of the servers in the
> map, however that would cause clients in Texas to exhibit the same
> behavior that clients in Minneapolis are.
> 
> Can somebody explain to me whether I'm not the right track in
> understanding the problem, with regards to the comparison not being
> fine-grained enough for the client to make the correct choice?  Or,
> might there be something else going on?  I previously had assumed that
> the behavior I was seeing was related to the bug that caused the
> client to always mount the first-listed server, but I no longer think
> that's the case.

Yes. This behavior is not what I'm after. The Solaris case is what I
want it to do.

Please bear in mind that this is still fairly new code and a number of
problems have been identified and solved so far.

I'll have a look at the state of the patches that Jeff has in release 12
and the code and see what I can find out.

You'll need to bear with me as I'm way busy atm.

Ian

Re: Replicated hosts in autofs v4.

[Jeff Moyer]

==> Regarding Re: [autofs] Replicated hosts in autofs v4.; Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> adds:

vda> On Sunday 10 October 2004 19:18, Jeremy Rosengren wrote:
>> I've previously posted about this problem, but I still haven't found an
>> answer, although I think I better understand what's happening.
>> 
>> We have an automount map called "auto_group" that looks like this:
>> 
>> icdes ausfiler,minnfiler:/vol/vol0/data/group/&
>> 
>> ausfiler lives in Texas, minnfiler lives in Minnesota.  Between ausfiler
>> and minnfiler lies a T1 line.
>> 
>> On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3
>> client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts
ausfiler:/vol/vol0/data/group/icdes.  No matter what...it always
>> mounts ausfiler.  The problem is that ausfiler is on the other end of a
>> T1 from the client, whereas minnfiler is on the same subnet, on the same
>> network switch.

vda> Your config does not let the system know that you prefer minnfiler
vda> over ausfiler if both are available.

>> I filed a support ticket with RedHat Global support, and an engineer
>> there confirmed that the client does an RPC call with a .1 second
>> timeout.  However, the client's behavior indicates to me that the
>> comparison being made in the reponse times between the replicated mount
>> servers isn't fine-grained enough to make the correct choice.  A
>> coworker made the comment that our T1 line could be "too robust", making
>> the server farther away appear to be good enough for the client's
>> purposes.
>> 
>> Solaris doesn't seem to have any issues with doing this properly.  A

vda> Prolly Solaris does not use first server from the set of those which
vda> answered, it uses server which _answers first_.  -- vda

Not true.  Solaris behaves quite as you would expect in these cases:

 Server proximity takes priority in the  selection  process.
     In  the  example  above,  if the server delta is on the same
     network segment as the client, but the others  are  on  dif-
     ferent  network  segments,  then delta will be selected; the
     weighting value is ignored.  The weighting has  effect  only
     when selecting between servers with the same network proxim-
     ity.

Above taken from the Solaris automount(1m) man page.

-Jeff

Re: Replicated hosts in autofs v4.

[Ian Kent]

On Mon, 11 Oct 2004, Jeff Moyer wrote:

> 
> Not true.  Solaris behaves quite as you would expect in these cases:
> 
>  Server proximity takes priority in the  selection  process.
>      In  the  example  above,  if the server delta is on the same
>      network segment as the client, but the others  are  on  dif-
>      ferent  network  segments,  then delta will be selected; the
>      weighting value is ignored.  The weighting has  effect  only
>      when selecting between servers with the same network proxim-
>      ity.

Mike, can we define "proximity" and "same network segment" as used in the 
Solaris automounter please (if that's possible).

Ian

Re: Replicated hosts in autofs v4.

[Jim Carter]

On Tue, 12 Oct 2004, Ian Kent wrote:
> On Mon, 11 Oct 2004, Jeff Moyer wrote:
> >  Server proximity takes priority in the  selection  process.
> 
> Mike, can we define "proximity" and "same network segment" as used in the 
> Solaris automounter please (if that's possible).

The desired behavior (and I think this is what our Sun boxes actually do)  
is that if the client has an interface such that (clientIP & netmask) ==
(serverIP & netmask) (using the client's same netmask for both), then that
server is preferred because the traffic goes direct, not through a router.  
The client might have multiple interfaces.

If there are multiple servers and one is down, another can be tried after a
brief timeout.  But what really interests me is hot failover:  you already
mounted server "A", which dies.  The automounter is magically aware of
this, and mounts server "B".  Can of worms: Processes with open files
(readonly of course) in the dead filesystem, you would like to transfer
them transparently to presumably identical files from the other server. 
In a map row with multiple servers, all share the same mount point, so how
do you dispose of the corpse so you can mount the other server on the 
same-named mount point?  None of this will be easy.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Replicated hosts in autofs v4.

[Mike Waychison]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jim Carter wrote:
> On Tue, 12 Oct 2004, Ian Kent wrote:
> 
>>On Mon, 11 Oct 2004, Jeff Moyer wrote:
>>
>>> Server proximity takes priority in the  selection  process.
>>
>>Mike, can we define "proximity" and "same network segment" as used in the 
>>Solaris automounter please (if that's possible).
> 
> 
> The desired behavior (and I think this is what our Sun boxes actually do)  
> is that if the client has an interface such that (clientIP & netmask) ==
> (serverIP & netmask) (using the client's same netmask for both), then that
> server is preferred because the traffic goes direct, not through a router.  
> The client might have multiple interfaces.
> 

I believe this is the right interpretation of the 'weight override' bit
that Jeff posted from the Solaris automount(1m) manpage.

However, I believe that algorithm uses some sort of traceroute/rpcping
to figure out distances and weights are used to prioritize given equal
distances.  I'll ask around to see what the actual method used is. (a
tcpdump of this may prove useful ;)

> If there are multiple servers and one is down, another can be tried after a
> brief timeout.  But what really interests me is hot failover:  you already
> mounted server "A", which dies.  The automounter is magically aware of
> this, and mounts server "B".  Can of worms: Processes with open files
> (readonly of course) in the dead filesystem, you would like to transfer
> them transparently to presumably identical files from the other server. 
> In a map row with multiple servers, all share the same mount point, so how
> do you dispose of the corpse so you can mount the other server on the 
> same-named mount point?  None of this will be easy.
> 

Note, this 'hot failover' functionality is only prescribed to work when
the filesystem is mounted _read only_.  I believe this is handled by the
NFS subsystem proper as the stateless nature of NFS2/3 would make this
feasible for the read-only case.  Again, I'll ask around to see how this
is handled and will post back when I get an answer.

- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE:  The opinions expressed in this email are held by me,
and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBa56ddQs4kOxk3/MRApIMAJ47NUyh7yoZucN8YKXvtu4AnR9JTQCgl7Ny
99axGIQugUaU3fBuggDb0iQ=
=0gDc
-----END PGP SIGNATURE-----

Re: Replicated hosts in autofs v4.

[raven]

On Tue, 12 Oct 2004, Mike Waychison wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Jim Carter wrote:
> > On Tue, 12 Oct 2004, Ian Kent wrote:
> > 
> >>On Mon, 11 Oct 2004, Jeff Moyer wrote:
> >>
> >>> Server proximity takes priority in the  selection  process.
> >>
> >>Mike, can we define "proximity" and "same network segment" as used in the 
> >>Solaris automounter please (if that's possible).
> > 
> > 
> > The desired behavior (and I think this is what our Sun boxes actually do)  
> > is that if the client has an interface such that (clientIP & netmask) ==
> > (serverIP & netmask) (using the client's same netmask for both), then that
> > server is preferred because the traffic goes direct, not through a router.  
> > The client might have multiple interfaces.
> > 
> 
> I believe this is the right interpretation of the 'weight override' bit
> that Jeff posted from the Solaris automount(1m) manpage.
> 
> However, I believe that algorithm uses some sort of traceroute/rpcping
> to figure out distances and weights are used to prioritize given equal
> distances.  I'll ask around to see what the actual method used is. (a
> tcpdump of this may prove useful ;)

OK. The traceroute idea is good but we have had people with port blocking 
so this would have to be quite smart.

Ian

Re: Replicated hosts in autofs v4.

[raven]

On Mon, 11 Oct 2004, Jim Carter wrote:

> On Tue, 12 Oct 2004, Ian Kent wrote:
> > On Mon, 11 Oct 2004, Jeff Moyer wrote:
> > >  Server proximity takes priority in the  selection  process.
> > 
> > Mike, can we define "proximity" and "same network segment" as used in the 
> > Solaris automounter please (if that's possible).
> 
> The desired behavior (and I think this is what our Sun boxes actually do)  
> is that if the client has an interface such that (clientIP & netmask) ==
> (serverIP & netmask) (using the client's same netmask for both), then that
> server is preferred because the traffic goes direct, not through a router.  
> The client might have multiple interfaces.
>

Yes, I thought of that but it occured to me that we may not want to use a 
server on the the same subnet. For example, on a multiple subnet gigabit 
network we might want to find the least loaded server. So this approach 
might be unreliable.

> If there are multiple servers and one is down, another can be tried after a
> brief timeout.  But what really interests me is hot failover:  you already
> mounted server "A", which dies.  The automounter is magically aware of
> this, and mounts server "B".  Can of worms: Processes with open files
> (readonly of course) in the dead filesystem, you would like to transfer
> them transparently to presumably identical files from the other server. 
> In a map row with multiple servers, all share the same mount point, so how
> do you dispose of the corpse so you can mount the other server on the 
> same-named mount point?  None of this will be easy.

Mmmm. NFS kernel land I think.
We'll have to be extra polite to the NFS gents!

Ian