Re: ARP flux vs. weak/strong ES model

Re: ARP flux vs. weak/strong ES model

[Lewis G Rosenthal]

Hi, Erik...

Erik Auerswald wrote:
> Hi,
>
> On 2/14/19 17:21, Grant Taylor wrote:
>> On 02/14/2019 01:31 AM, Erik Auerswald wrote:

<huge snip>

> Linux's cavalier default behaviour in answering ARP requests might be 
> motivated by the weak ES model in helping other systems on the LAN 
> reach the Linux server, even if they use an IP address assigned to 
> another LAN the Linux server is connected to. Thus the problem of "ARP 
> flux" is probably closely related to how Linux implements the weak ES 
> model, but not necessarily to the weak ES model itself as described in 
> RFC 1122.
>
> Thus I argue that using "ARP flux" to describe the ARP problem 
> observed with Linux is preferable to attributing the problems to 
> Linux's implementation of the weak ES model.
>
> Please note that I have not searched for the origin or an 
> authoritative source on "ARP flux", and cannot guarantee that it is 
> indeed consistently used to describe the aforementioned problem. But 
> it did turn up on related web searches, and seems to directly refer to 
> the ARP problem at hand.
>
> Sorry for being pedantic.
>

Thanks so much for a very cogent explanation of the difference. I'd been 
meaning to do some digging since spying this thread scrolling by. One of 
the things I like best about this list is the way it makes me scratch my 
head. Usually, someone else comes up with a good answer before I'm done 
scratching, which is another thing I like. :-)

Some interesting discussion of the Linux ARP flux problem, here (2.1.4 
et seq):

http://linux-ip.net/html/ether-arp.html

Linked from there, and perhaps relevant to the original problem:

http://ja.ssi.bg/#iparp

(I guess I should probably push those to the original thread.)

-- 
Lewis
-------------------------------------------------------------
Lewis G Rosenthal, CNA, CLP, CLE, CWTS, EA
Rosenthal & Rosenthal, LLC                www.2rosenthals.com
visit my IT blog                www.2rosenthals.net/wordpress
-------------------------------------------------------------

Re: ARP flux vs. weak/strong ES model

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 5972 bytes --]

On 2/15/19 1:11 PM, Erik Auerswald wrote:
> The first time I read about this, the issue was framed in the weak 
> end-system model vs. strong end-system model terminology. Thus I have 
> used that terminology in the past whenever a related problem occurred.

The earliest reference I could find to "ARP flux" was in the Guide to IP 
Layer Network Administration document, with references going back to 2003.

I get the impression that the "ARP flux" nomenclature may have 
originated from the Linux community and grown out from there.

Many, but not all, of the references to "weak vs strong end system" are 
older than that, or by people that learned about things (likely) pre-2000.

I think that "weak vs strong end system" and "ARP flux" are functionally 
the same thing.  Or at least the same concept told from opposing ends.

It seems to me that Linux's (default) "weak end system" model is the 
cause of "ARP flux" (in Linux).

> I have seen it just recently, and do not remember having read it before. 
> It turned up while searching the web for ARP related Linux sysctl settings.

Seeing as how the "ARP flux" phrase seems (to me) to be Linux centric. 
It makes sense that it would turn up when searching for articles about 
ARP issues on Linux.

> To turn this around, searching the web for "ARP flux" seems to turn up 
> better results regarding the original poster's issue than searching for 
> "weak end-system model". YMMV.

I've frequently noticed that older concepts are frequently harder to 
find than newer concepts.  I think this is likely somewhat natural based 
on how recent a concept is.

There is probably a health portion of things being better documented and 
/ or communicated post-2000, particularly with availability of web / 
Internet things.

> TL;DR I prefer precision when talking about technical problems. :-)

I tend to agree.

But I don't think "ARP flux" is a precise reason.  I think the "weak vs 
strong end system" is the reason behind the "ARP flux" symptom.

> I'd like to take a closer look at the problem at hand to better 
> understand it and the terminology used to describe it.
> 
> I'd say the distinction between weak and strong end-system (ES) model is 
> related to the problem at hand, but "ARP flux" is a distinct issue not 
> necessarily observed with a host implementing the weak (ES) model.

Do you know of any examples where the "ARP flux" /symptom/ was observed 
on a "strong end system"?

> RFC 1122 continues to state that "[The weak ES] model [...] is necessary 
> for hosts that have embedded gateway functionality."

I disagree with that assertion.  (Not your recounting of it.)

> A gateway (or router) needs to accept IP packets not addressed to the 
> receiving interface in order to forward them. But most commercial 
> routers will not answer an ARP request ingressing on the wrong 
> interface, unless Proxy-ARP is activated. By default, Linux does answer 
> such ARP requests.

I believe the MUST vs MUST NOT text is primarily about traffic to the 
end system, not traffic passing through the system.  Particularly how 
the system's IP stack responds in relation to IPs bound locally.

I think that routing / forwarding is decidedly different.  Yes, a router 
must allow traffic in that is not destined to the router.  (Assuming 
that we're talking about a forwarding interface and not a management 
interface.)  That's the router's job.

But the router does not need to respond to ARP requests from one 
interface for IPs on a different interface.

> This answer to ARP requests arriving at the wrong interface is the root 
> cause for most problems commonly subsumed under "Linux implements the 
> weak ES model". But those problems do not occur with other weak ES model 
> implementing systems, e.g. Extreme Networks EXOS switches.

Each IP stack is different.  I wouldn't take the difference in behavior 
of the Linux IP stack and EXOS's IP stack as definitive behavior.

> Additionally, changing Linux behaviour (via sysctl) to not answer those 
> ARP requests does not change the weak ES model applicability to Linux 
> (i.e. IP packets delivered inside an Ethernet frame addressed to an 
> interface MAC of a Linux system are accepted even if they are addressed 
> to another IP address of the Linux host; packet forwarding is not 
> affected either).
> 
> Linux's cavalier default behaviour in answering ARP requests might be 
> motivated by the weak ES model in helping other systems on the LAN reach 
> the Linux server, even if they use an IP address assigned to another LAN 
> the Linux server is connected to. Thus the problem of "ARP flux" is 
> probably closely related to how Linux implements the weak ES model, but 
> not necessarily to the weak ES model itself as described in RFC 1122.

I feel like "ARP flux" is a /symptom/ of "weak ES model".

Does a fever cause a cold?  Or can a cold cause a fever?  You can buy 
both cold medicine and fever reducers.  ;-)

(I know it's not a perfect analogy.  But I suspect you get my point.)

> Thus I argue that using "ARP flux" to describe the ARP problem observed 
> with Linux is preferable to attributing the problems to Linux's 
> implementation of the weak ES model.

You logic makes sense.  I choose to think something different.

> Please note that I have not searched for the origin or an authoritative 
> source on "ARP flux", and cannot guarantee that it is indeed 
> consistently used to describe the aforementioned problem. But it did 
> turn up on related web searches, and seems to directly refer to the ARP 
> problem at hand.

See my comments above.

> Sorry for being pedantic.

Apology returned as unnecessary.

I often find that I need to be pedantic when making sure I understand 
something, and / or communicating minutia details about it with someone.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: ARP flux vs. weak/strong ES model

[Erik Auerswald]

Ji,

On 2/17/19 06:10, Grant Taylor wrote:
> On 2/15/19 1:11 PM, Erik Auerswald wrote:
> [...]
> The earliest reference I could find to "ARP flux" was in the Guide to IP 
> Layer Network Administration document, with references going back to 2003.
> 
> I get the impression that the "ARP flux" nomenclature may have 
> originated from the Linux community and grown out from there.

Perhaps because it is a Linux only thing? That is just a guess.

>> [...]
>> I'd say the distinction between weak and strong end-system (ES) model 
>> is related to the problem at hand, but "ARP flux" is a distinct issue 
>> not necessarily observed with a host implementing the weak (ES) model.
> 
> Do you know of any examples where the "ARP flux" /symptom/ was observed 
> on a "strong end system"?

I have only ever seen it on Linux.

>> RFC 1122 continues to state that "[The weak ES] model [...] is 
>> necessary for hosts that have embedded gateway functionality."
> 
> I disagree with that assertion.  (Not your recounting of it.)

I'd say that assertion is often correct, but need not be.

I think that this assertion motivates looking at non-Linux systems, 
especially traditional routers, and if they act as weak or strong 
end-systems. And then look at their ARP handling, excluding Proxy ARP.

>> A gateway (or router) needs to accept IP packets not addressed to the 
>> receiving interface in order to forward them. But most commercial 
>> routers will not answer an ARP request ingressing on the wrong 
>> interface, unless Proxy-ARP is activated. By default, Linux does 
>> answer such ARP requests.
> 
> I believe the MUST vs MUST NOT text is primarily about traffic to the 
> end system, not traffic passing through the system.  Particularly how 
> the system's IP stack responds in relation to IPs bound locally.
> 
> I think that routing / forwarding is decidedly different.  Yes, a router 
> must allow traffic in that is not destined to the router.  (Assuming 
> that we're talking about a forwarding interface and not a management 
> interface.)  That's the router's job.

Yes, a system dedicated to packet forwarding is different. Often it is 
an end-system as well for management purposes, sometimes for additional 
services (inlcuding BGP). RFC 1122 looks at IP hosts, not necessarily 
dedicated gateways, but includes hosts that act as gateways, too. Thus I 
do not say that RFC 1122 is necessarily pertinent to dedicated routers, 
but often even dedicated routers act like end-systems with embedded 
gateway functionality.

> But the router does not need to respond to ARP requests from one 
> interface for IPs on a different interface.

Exactly. Unless Proxy ARP is active, any "traditional" router I know 
answers ARP requests only if they arrive at the interface configured 
with the IP address in the ARP request.

Again, any "traditional" router accepts IP packets directed to any of 
its interface IPs irrespective of the ingress interface. That is the 
basis for using a loopback address for router management or BGP 
sessions. In that case a router acts as an end-system as well.

The above can often be changed via configuration, to separate management 
interfaces from forwarding interfaces.

>> This answer to ARP requests arriving at the wrong interface is the 
>> root cause for most problems commonly subsumed under "Linux implements 
>> the weak ES model". But those problems do not occur with other weak ES 
>> model implementing systems, e.g. Extreme Networks EXOS switches.
> 
> Each IP stack is different.  I wouldn't take the difference in behavior 
> of the Linux IP stack and EXOS's IP stack as definitive behavior.

But the weak vs. strong ES model is independent of IP stacks. It is a 
description of behaviour, and can be used as guidance when implementing 
an IP stack.

The two examples are just that, examples, nothing more. Cisco IOS (with 
deactivated Proxy ARP) is another example for weak ES model without ARP 
flux. But just one example (e.g. of Extreme or Cisco) suffices to show 
that weak ES model does not imply ARP flux. The Linux example shows that 
weak ES model and ARP flux can occur together.

>> Additionally, changing Linux behaviour (via sysctl) to not answer 
>> those ARP requests does not change the weak ES model applicability to 
>> Linux (i.e. IP packets delivered inside an Ethernet frame addressed to 
>> an interface MAC of a Linux system are accepted even if they are 
>> addressed to another IP address of the Linux host; packet forwarding 
>> is not affected either).
>>
>> Linux's cavalier default behaviour in answering ARP requests might be 
>> motivated by the weak ES model in helping other systems on the LAN 
>> reach the Linux server, even if they use an IP address assigned to 
>> another LAN the Linux server is connected to. Thus the problem of "ARP 
>> flux" is probably closely related to how Linux implements the weak ES 
>> model, but not necessarily to the weak ES model itself as described in 
>> RFC 1122.
> 
> I feel like "ARP flux" is a /symptom/ of "weak ES model".

I'd say it is somewhat independent of the weak ES model. It is a symptom 
of the Linux IP stack. That IP stack may be built around weak ES model 
ideas. Other IP stacks adhere to the weak ES model as well without 
exhibiting ARP flux. Thus I do not accept that "ARP flux" is a symptom 
(or necessary result) of "weak ES model".

But it seems to me that combining strong ES model with ARP flux does not 
make sense. As such I do see some relation between ARP flux and weak ES 
model.

> [...]
>> Thus I argue that using "ARP flux" to describe the ARP problem 
>> observed with Linux is preferable to attributing the problems to 
>> Linux's implementation of the weak ES model.
> 
> You logic makes sense.  I choose to think something different.

Then we have to agree to disagree.

Thanks,
Erik

Re: ARP flux vs. weak/strong ES model

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 6046 bytes --]

On 2/17/19 5:37 PM, Erik Auerswald wrote:
> I have only ever seen it on Linux.

Likewise.

> I think that this assertion motivates looking at non-Linux systems, 
> especially traditional routers, and if they act as weak or strong 
> end-systems. And then look at their ARP handling, excluding Proxy ARP.

Before looking at other systems, I'd want to step back and think how 
weak vs strong end-systems /should/ behave regarding ARP flux.

Aside:  I think of Proxy ARP as a form of routing.  After all, the Proxy 
ARP router, is doing exactly what it would do to route a packet if it 
had naturally come to it / it's MAC address.  All Proxy ARP is doing is 
responding to ARP requests for things on the other side of the router 
such that the packet does come to the router.

> Yes, a system dedicated to packet forwarding is different. Often it is 
> an end-system as well for management purposes, sometimes for additional 
> services (inlcuding BGP). RFC 1122 looks at IP hosts, not necessarily 
> dedicated gateways, but includes hosts that act as gateways, too. Thus I 
> do not say that RFC 1122 is necessarily pertinent to dedicated routers, 
> but often even dedicated routers act like end-systems with embedded 
> gateway functionality.

Fair.

> Exactly. Unless Proxy ARP is active, any "traditional" router I know 
> answers ARP requests only if they arrive at the interface configured 
> with the IP address in the ARP request.

ACK

> Again, any "traditional" router accepts IP packets directed to any of 
> its interface IPs irrespective of the ingress interface. That is the 
> basis for using a loopback address for router management or BGP 
> sessions. In that case a router acts as an end-system as well.

That's not entirely true.  Especially when filtering / firewalling is in 
place to only allow traffic from specific interfaces.

I've also viewed that as the traffic would be routed through the device 
to the proper interface which would then process it accordingly.  In, 
over / through and then up the IP stack instead of in and directly up 
the IP stack.

> The above can often be changed via configuration, to separate management 
> interfaces from forwarding interfaces.

I view the fact that the router is also an end system for management 
purposes as independent of it's routing function.

> But the weak vs. strong ES model is independent of IP stacks. It is a 
> description of behaviour, and can be used as guidance when implementing 
> an IP stack.

Okay.

> The two examples are just that, examples, nothing more. Cisco IOS (with 
> deactivated Proxy ARP) is another example for weak ES model without ARP 
> flux. But just one example (e.g. of Extreme or Cisco) suffices to show 
> that weak ES model does not imply ARP flux. The Linux example shows that 
> weak ES model and ARP flux can occur together.

Let's back up and discuss what is actually allowing ARP flux to happen.

As I understand it, the /flux/ comes from the fact that the MAC address 
that ARPing hosts get replies from changes and fluctuates.  Hence the name.

It's my understanding that this happens because Linux does not filter 
(in any meaning of the word) incoming ARP requests (or outgoing replies) 
based on the physical interface.  This is especially true when you have 
multiple interfaces in the same broadcast domain.

Aside:  The last time I tried to put two interfaces in the same subnet 
and connect them to the same broadcast domain on a Cisco, it would not 
allow me to do so.

> I'd say it is somewhat independent of the weak ES model. It is a symptom 
> of the Linux IP stack. That IP stack may be built around weak ES model 
> ideas. Other IP stacks adhere to the weak ES model as well without 
> exhibiting ARP flux.

Sorry for being pedantic, but I think we need to clearly define the 
configuration and behavior that we're discussing.

I say this because I think that "ARP flux" is a symptom of having a 
Linux box with two interfaces in the same broadcast domain, thus able to 
hear the same ARP request and that the flux comes from the ensuing race 
conditions as to which interface will be processed -and- reply first.

I feel like this same scenario is seldom played out in traditional 
network gear.  And if we want to have the discussion about this, we 
should configure said gear comparably and test how it behaves.

I will also state that Linux may likely respond to ARP requests from an 
inside interface for IPs on the outside interface.  But in such a 
scenario, there is only one interface connected to the broadcast domain, 
thus there is nothing to flux over as it will always be the single 
possible interface.

So, let's define what the connections are, and how things are configured.

I'm stating two interfaces connected to the same broadcast domain, each 
with IPs in different subnets.  (Thus the broadcast domain is overloaded 
and has multiple subnets on it.)  I think there is a reasonable chance 
that the ARP flux symptoms can occur in this configuration.

I'm thinking Linux /kernel/ default (no distro sysctl modifications or 
kernel compilation tweaks).  I'm also thinking Proxy ARP is disabled.

Do you agree?  Or do you want to alter the configuration?

> Thus I do not accept that "ARP flux" is a symptom (or necessary result) 
> of "weak ES model".

Okay.

> But it seems to me that combining strong ES model with ARP flux does not 
> make sense. As such I do see some relation between ARP flux and weak ES 
> model.

Okay.

> Then we have to agree to disagree.

I don't see a problem with that.  It's less than ideal, but it is a 
perfectly valid outcome.

That being said, you do have me questioning things.  At the moment, I'm 
sticking with what I've thought for years.  But I am interested in 
continuing the conversation and learning, what ever the lesson may be.

> Thanks,

You're welcome.

Thank you.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: ARP flux vs. weak/strong ES model

[Erik Auerswald]

Hi,

On 2/18/19 03:18, Grant Taylor wrote:
> On 2/17/19 5:37 PM, Erik Auerswald wrote:
>> I have only ever seen it on Linux.
> 
> Likewise.
> 
>> I think that this assertion motivates looking at non-Linux systems, 
>> especially traditional routers, and if they act as weak or strong 
>> end-systems. And then look at their ARP handling, excluding Proxy ARP.
> 
> Before looking at other systems, I'd want to step back and think how 
> weak vs strong end-systems /should/ behave regarding ARP flux.

As I see it, strong ES model should preclude ARP flux, because an ES 
first telling "use this MAC to send to my IP X", but then discarding the 
packet sent based on that information, seems nonsensical.

In the weak ES model, telling another ES to send to MAC Y to deliver to 
IP X, although the interface with MAC Y uses IP Z≠X seems OK, but not 
mandatory. To me this feels much like the use of Proxy ARP to work 
around misconfigured end-systems. Ignoring ARP requests on all 
interfaces but the one with the asked for IP seems OK, too.

> Aside:  I think of Proxy ARP as a form of routing.

Freely combining MAC and IP addresses in ARP replies looks quite similar 
to Proxy ARP to me.

> After all, the Proxy 
> ARP router, is doing exactly what it would do to route a packet if it 
> had naturally come to it / it's MAC address.  All Proxy ARP is doing is 
> responding to ARP requests for things on the other side of the router 
> such that the packet does come to the router.

That is correct, but Proxy ARP will result in a router answering an ARP 
request that was not for the receiving interface, thus it could be 
confused with ARP flux in a test.

>> [...]
>> Again, any "traditional" router accepts IP packets directed to any of 
>> its interface IPs irrespective of the ingress interface. That is the 
>> basis for using a loopback address for router management or BGP 
>> sessions. In that case a router acts as an end-system as well.
> 
> That's not entirely true.  Especially when filtering / firewalling is in 
> place to only allow traffic from specific interfaces.

Filtering / firewalling can be in effect in Linux as well, including for 
Ethernet (ebtables). That would most likely affect ARP flux and weak ES 
model behaviour as well, depending on rule set.

> I've also viewed that as the traffic would be routed through the device 
> to the proper interface which would then process it accordingly.  In, 
> over / through and then up the IP stack instead of in and directly up 
> the IP stack.

It is not observably "routed" in that all routing actions (L2 rewrite, 
TTL decrement) usually happen when sending the packet _out_ the egress 
interface.

>> [...]
> Let's back up and discuss what is actually allowing ARP flux to happen.
> 
> As I understand it, the /flux/ comes from the fact that the MAC address 
> that ARPing hosts get replies from changes and fluctuates.  Hence the name.
> 
> It's my understanding that this happens because Linux does not filter 
> (in any meaning of the word) incoming ARP requests (or outgoing replies) 
> based on the physical interface.  This is especially true when you have 
> multiple interfaces in the same broadcast domain.

That is a logical explanation of the name "ARP flux".

> Aside:  The last time I tried to put two interfaces in the same subnet 
> and connect them to the same broadcast domain on a Cisco, it would not 
> allow me to do so.

Correct. Huawei VRP allows a special case (a loopback interface with a 
/32 IP inside an IP subnet with shorter prefix active on another 
interface, including Ethernet interfaces), but I have not yet found time 
to thoroughly test the behaviour of that. Other networking equipment I 
used did not allow two interfaces in the same (or overlapping) subnet(s).

>> I'd say it is somewhat independent of the weak ES model. It is a 
>> symptom of the Linux IP stack. That IP stack may be built around weak 
>> ES model ideas. Other IP stacks adhere to the weak ES model as well 
>> without exhibiting ARP flux.
> 
> Sorry for being pedantic, but I think we need to clearly define the 
> configuration and behavior that we're discussing.
> 
> I say this because I think that "ARP flux" is a symptom of having a 
> Linux box with two interfaces in the same broadcast domain, thus able to 
> hear the same ARP request and that the flux comes from the ensuing race 
> conditions as to which interface will be processed -and- reply first.
> 
> I feel like this same scenario is seldom played out in traditional 
> network gear.  And if we want to have the discussion about this, we 
> should configure said gear comparably and test how it behaves.
> 
> I will also state that Linux may likely respond to ARP requests from an 
> inside interface for IPs on the outside interface.  But in such a 
> scenario, there is only one interface connected to the broadcast domain, 
> thus there is nothing to flux over as it will always be the single 
> possible interface.
> 
> So, let's define what the connections are, and how things are configured.
> 
> I'm stating two interfaces connected to the same broadcast domain, each 
> with IPs in different subnets.  (Thus the broadcast domain is overloaded 
> and has multiple subnets on it.)  I think there is a reasonable chance 
> that the ARP flux symptoms can occur in this configuration.
> 
> I'm thinking Linux /kernel/ default (no distro sysctl modifications or 
> kernel compilation tweaks).  I'm also thinking Proxy ARP is disabled.
> 
> Do you agree?  Or do you want to alter the configuration?

I want to extend that scenario to include two interfaces A and B with 
different IP addresses connected to two separate broadcast domains. 
While that does not result in fluctuating ARP replies (ARP flux), it 
does result in ARP replies combining MAC of interface A with IP of 
interface B.

Both scenarios (two interfaces connected to one broadcast domain, two 
interfaces connected to separate broadcast domains) show symptoms of the 
same underlying cause. The name ARP _flux_ is more fitting to the first 
scenario, the second could be better described as "ARP confusion" (I 
made that name up just now).

> [...]
> That being said, you do have me questioning things.  At the moment, I'm 
> sticking with what I've thought for years.  But I am interested in 
> continuing the conversation and learning, what ever the lesson may be.

Likewise.

Best regards,
Erik