Re: [PATCH] strict interface arp patch for Linux 2.4.2

Re: [PATCH] strict interface arp patch for Linux 2.4.2

[Allen Lau]

Julian Anastasov <ja@ssi.bg> on 09/20/2001 06:19:04 PM

To:   Allen Lau/Raleigh/IBM@IBMUS
cc:   linux-kernel <linux-kernel@vger.kernel.org>
Subject:  Re: [PATCH] strict interface arp patch for Linux 2.4.2




     Hello,

On Thu, 20 Sep 2001, Allen Lau wrote:

>> I want to bring your attention to a Linux ARP patch we plan to use for
load balancing and server
>> clustering.  The available arp filter and hidden patch are not
completely satisfactory. The following

>    Can you explain what setup can't be build using arp_filter,
> hidden or rp_filter and particulary where "hidden" fails for clusters,
> it is very interesting?

The function provided by "hidden" is a necessary element but not
sufficient. I suppose arp_filter
has to be combined with the hidden patch. I dislike hiding an interface,
and the hidden patch is no
longer included in the major distributions which makes delivering a
solution difficult.

In this example, 1.1.1.11 2.2.2.22 are virtual ip's which can float between
boxes and interfaces.
I want virtual ip 1.1.1.11 to be known to clients through eth0 node1
(2.2.2.22 eth1 node2). Node1
can redirect 1.1.1.11 traffic to eth0 node2.
                          node1                                 node2
                      lo:1 1.1.1.11                          lo:1 1.1.1.11
                      lo:2 2.2.2.22                          lo:2 2.2.2.22

              eth0   1.1.1.1     eth1 2.2.2.1         eth0  1.1.1.2   eth1
2.2.2.2
              eth0:1 1.1.1.11
eth1:1 2.2.2.22

On node1, does hiding loopback hide 1.1.1.11? and can 2.2.2.1 be advertised
as source ip in arp
requests from eth0?

>> To generalize, each real server may have multiple nic's of different
types. The task becomes one of
?> maintaining strict identity of each of the real and virtual ip
addresses.  The Linux ARP has the
>> following behaviors which are problematic for maintaining strict
interface identify.
>>
>>    1) box responds to arp on all interfaces on the same wire for an IP
address (arp race)

>    rp_filter and arp_filter can help here but this "arp race" is not
> an evil in some setups

I believe "arp race" may cause purges in a switched network. It certainly
is counter to maintaining
strict interface identity.

Regards
Allen Lau
--
Julian Anastasov <ja@ssi.bg>

Re: [PATCH] strict interface arp patch for Linux 2.4.2

[Allen Lau]

---------------------- Forwarded by Allen Lau/Raleigh/IBM on 09/24/2001
06:03 PM ---------------------------

Allen Lau
09/24/2001 01:09 PM

To:   Julian Anastasov <ja@ssi.bg>
cc:   inux-kernel@vger.kernel.org
From: Allen Lau/Raleigh/IBM@IBMUS
Subject:  Re: [PATCH] strict interface arp patch for Linux 2.4.2  (Document
      link: Allen Lau)

     Hello,

> - you make the assumption that all primary local IP addresses are
> configured on ARP devices. By this way you can't combine virtual
> network (with netsize < 32 bits on device lo) with your need for
> strict traffic paths. To allow this you have to clear the strict
> arp flag from the ARP devices. We don't talk about solutions where
> one host needs both primary and secondary network from shared
> addresses (both on device lo). This can be solved only with the
> route's noarp flag (ip addr add NET1/24 dev lo hidden,
> ip addr add NET2/24 dev lo), even hidden can't help here.

Julian, thanks for sharing the background info.
I assume that strict_interface_arp be enabled only for interfaces
that require it (e.g. for bandwidth control). It does not require loopback
to be hidden. You may have other interfaces (different ip subnets) that
does unrestricted operations.

> - something new: should the strict arp flag stop the "arp race" caused
> from the proxy_arp feature (i.e. to stop the proxy_arp feature)?

The arp patches all operate on the case of RTN_LOCAL (address in the box)
- proxy_arp should not be affected.

> > That's the "arp flux" problem which strict_interface_arp solves.
> > In Linux arp requests, it is possible for 2.2.2.1 eth1 node1 to be
advertised
> > as source by eth0 node1. For illustration, eth0 is 10/100M and eth1 is
gigabit.
> > It'll cause clients to redirect traffic for the gigabit to the 10/100M
port.

>    Well, you are talking about the case where someone binds to
> 2.2.2.1 and talks to 1.1.1.2. What to say, IMO the right solution is
> arp_solicit always to select the preferred source address from a new
> [0 -> target, neigh->dev] output route call. We know that the remote
> host should be able to receive our probes from many devices (if the
> IP traffic is also received there) but it helps to solve the "arp flux"
> problem and helps not to announce shared addresses in our probes.
> Then the filtering of the remote probes can be done in different ways.
> I don't know what the maintainers have against it. IMO, the route's
> noarp flag + arp_solicit always to use the prefsrc is the best we can
> do for 2.4.

Agreed. (strict_interface_arp touched arp_solicit and inet_addr_select for
that)

Regards
Allen Lau
--
Julian Anastasov <ja@ssi.bg>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] strict interface arp patch for Linux 2.4.2

[Julian Anastasov]

	Hello,

On Sat, 22 Sep 2001, Allen Lau wrote:

> > > In this example, 1.1.1.11 2.2.2.22 are virtual ip's which can float
> between
> > > boxes and interfaces.
> > > I want virtual ip 1.1.1.11 to be known to clients through eth0 node1
> > > (2.2.2.22 eth1 node2). Node1 can redirect 1.1.1.11 traffic to eth0
> node2.
> > >               node1                              node2
> > >           lo:1 1.1.1.11                       lo:1 1.1.1.11
> > >           lo:2 2.2.2.22                       lo:2 2.2.2.22
> > >
> > >   eth0   1.1.1.1    eth1 2.2.2.1      eth0 1.1.1.2    eth1 2.2.2.2
> > >   eth0:1 1.1.1.11                                   eth1:1 2.2.2.22
> > >
>
> arp_filter is not enough:
>    with arp_filter enabled, both node 1 and 2 will answer to "who is 1.1.1.11?".

	Agreed, arp_filter is not a solution for shared addresses.
I have never said such thing. You should assume that there is a reason
both hidden and arp_filter to exist in 2.2.14+ - they do different
things. You need only to mark {all,lo}/hidden on the both nodes. This is
enough, no matter what is the value of arp_filter and rp_filter.

> hidden patch + arp_filter:
>    If the hidden and arp_filter patches can be applied on top of each other(?),

	Yes, starting from hidden-2.4.5-1.diff (2.4.5 was the first 2.4
kernel with arp_filter support), may be your assumptions are based on
2.4.2 :)

>    one still has to be careful configuring the system.  In the example with
>    loopback hidden, you'll notice that no one will answer "who is 1.1.1.11?".

	May be because you have configured 1.1.1.11 both on lo:1 and
eth0:1. Agreed, it is dangerous. At least, the users have the power to
do advanced things without restrictions.

>    Further, arp_filter (which is based on routing) may have side effects from
>    asymmetric routing, multiple nic's on same subnet, and default routes.

	My opinion about all these flags and problems is that there
are many possible setups that need different solutions. All these flags
play different roles. The missing part from 2.4 (that you don't see in
distributions) is the solution for the ARP behavior when many hosts
share same addresses. All other problems you see can be solved from
the existing device flags and may be with route's noarp flag (plus changes
in arp_solicit).

	I already mentioned where the strict_interface_arp flag is
not good for clusters (may be it is good for other things but there
are already other flags for this):

- you make the assumption that all primary local IP addresses are
configured on ARP devices. By this way you can't combine virtual
network (with netsize < 32 bits on device lo) with your need for
strict traffic paths. To allow this you have to clear the strict
arp flag from the ARP devices. We don't talk about solutions where
one host needs both primary and secondary network from shared
addresses (both on device lo). This can be solved only with the
route's noarp flag (ip addr add NET1/24 dev lo hidden,
ip addr add NET2/24 dev lo), even hidden can't help here.

- something new: should the strict arp flag stop the "arp race" caused
from the proxy_arp feature (i.e. to stop the proxy_arp feature)?

> This example illustrates the problem and that the distributions does not have
> enough to solve. In addition, I need something simpler for field deployment.
> Strict_interface_arp I hope will provide the needed simplicity and control.

	Same with rp_filter+hidden

> >    The question is why you want 2.2.2.1 to be advertised as source
> > through eth0 ? May be your question is wrong and should contain eth1?
> > inet_select_addr() obviously selects source IP from the device you
> > request.
>
> That's the "arp flux" problem which strict_interface_arp solves.
> In Linux arp requests, it is possible for 2.2.2.1 eth1 node1 to be advertised
> as source by eth0 node1. For illustration, eth0 is 10/100M and eth1 is gigabit.
> It'll cause clients to redirect traffic for the gigabit to the 10/100M port.
> It does not appear rp_filter can solve this.

	Well, you are talking about the case where someone binds to
2.2.2.1 and talks to 1.1.1.2. What to say, IMO the right solution is
arp_solicit always to select the preferred source address from a new
[0 -> target, neigh->dev] output route call. We know that the remote
host should be able to receive our probes from many devices (if the
IP traffic is also received there) but it helps to solve the "arp flux"
problem and helps not to announce shared addresses in our probes.
Then the filtering of the remote probes can be done in different ways.
I don't know what the maintainers have against it. IMO, the route's
noarp flag + arp_solicit always to use the prefsrc is the best we can
do for 2.4.

> Regards
> Allen Lau


Regards

--
Julian Anastasov <ja@ssi.bg>

Re: [PATCH] strict interface arp patch for Linux 2.4.2

[Allen Lau]

     Hello,

> > In this example, 1.1.1.11 2.2.2.22 are virtual ip's which can float
between
> > boxes and interfaces.
> > I want virtual ip 1.1.1.11 to be known to clients through eth0 node1
> > (2.2.2.22 eth1 node2). Node1 can redirect 1.1.1.11 traffic to eth0
node2.
> >               node1                              node2
> >           lo:1 1.1.1.11                       lo:1 1.1.1.11
> >           lo:2 2.2.2.22                       lo:2 2.2.2.22
> >
> >   eth0   1.1.1.1    eth1 2.2.2.1      eth0 1.1.1.2    eth1 2.2.2.2
> >   eth0:1 1.1.1.11                                   eth1:1 2.2.2.22
> >

arp_filter is not enough:
   with arp_filter enabled, both node 1 and 2 will answer to "who is 1.1.1.11?".

hidden patch + arp_filter:
   If the hidden and arp_filter patches can be applied on top of each other(?),
   one still has to be careful configuring the system.  In the example with
   loopback hidden, you'll notice that no one will answer "who is 1.1.1.11?".
   Further, arp_filter (which is based on routing) may have side effects from
   asymmetric routing, multiple nic's on same subnet, and default routes.

This example illustrates the problem and that the distributions does not have
enough to solve. In addition, I need something simpler for field deployment.
Strict_interface_arp I hope will provide the needed simplicity and control.

>    The question is why you want 2.2.2.1 to be advertised as source
> through eth0 ? May be your question is wrong and should contain eth1?
> inet_select_addr() obviously selects source IP from the device you
> request.

That's the "arp flux" problem which strict_interface_arp solves.
In Linux arp requests, it is possible for 2.2.2.1 eth1 node1 to be advertised
as source by eth0 node1. For illustration, eth0 is 10/100M and eth1 is gigabit.
It'll cause clients to redirect traffic for the gigabit to the 10/100M port.
It does not appear rp_filter can solve this.

Regards
Allen Lau
--
Julian Anastasov <ja@ssi.bg>

Re: [PATCH] strict interface arp patch for Linux 2.4.2

[Julian Anastasov]

	Hello,

On Thu, 20 Sep 2001, Allen Lau wrote:

> I want to bring your attention to a Linux ARP patch we plan to use for load balancing and server
> clustering.  The available arp filter and hidden patch are not completely satisfactory. The following

	Can you explain what setup can't be build using arp_filter,
hidden or rp_filter and particulary where "hidden" fails for clusters,
it is very interesting?

> To generalize, each real server may have multiple nic's of different types. The task becomes one of
> maintaining strict identity of each of the real and virtual ip addresses.  The Linux ARP has the
> following behaviors which are problematic for maintaining strict interface identify.
>
>    1) box responds to arp on all interfaces on the same wire for an IP address (arp race)

	rp_filter and arp_filter can help here but this "arp race" is not
an evil in some setups

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: [PATCH] strict interface arp patch for Linux 2.4.2

[David S. Miller]

   From: "Allen Lau" <pflau@us.ibm.com>
   Date: Thu, 20 Sep 2001 12:12:53 -0400

   The available arp filter and hidden patch are not completely satisfactory.

If you were to setup your routing correctly, it would be satisfactory.

Later,
David S. Miller
davem@redhat.com

[PATCH] strict interface arp patch for Linux 2.4.2

[Allen Lau]

I want to bring your attention to a Linux ARP patch we plan to use for load balancing and server
clustering.  The available arp filter and hidden patch are not completely satisfactory. The following
post by Julian Anastasov and Bernd Eckenfels described the virtual server clustering problem.

      http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.3/0188.html

To generalize, each real server may have multiple nic's of different types. The task becomes one of
maintaining strict identity of each of the real and virtual ip addresses.  The Linux ARP has the
following behaviors which are problematic for maintaining strict interface identify.

   1) box responds to arp on all interfaces on the same wire for an IP address (arp race)
   2) interface responds to arp requests for any of its IP addresses (loopback not hidden)
   3) box responds to arp request for downed interface IP address
   4) interface uses as source (in arp request) IP address from any other interface (arp flux)

The proposed strict interface arp patch tackles the strict interface identify problem by explicitly
binding arp request and response to the interface (i.e. an interface uses as source and responds to
ARPs for an IP address only if the address is configured on the interface).

The strict_interface_arp patch can be enabled for individual or all interfaces.
   1) to enable for eth0 dynamically
      [/root]# echo 1 > /proc/sys/net/ipv4/conf/eth0/strict_interface_arp

   2) to enable for all interfaces on boot using /etc/sysctl.conf
      [/root]# cat /etc/sysctl.conf
      # Enable strict_interface_arp :
      net.ipv4.conf.all.strict_interface_arp = 1

Your comments are sincerely appreciated.

Allen Lau
IBM Storage Network Division
Research Triangle Park, NC  27709
email : pflau@us.ibm.com

diff -Naur linux-2.4.2-2/Documentation/filesystems/proc.txt linux-2.4.2-2-strict_interface_arp/Documentation/filesystems/proc.txt
--- linux-2.4.2-2/Documentation/filesystems/proc.txt    Sat Sep  8 22:08:40 2001
+++ linux-2.4.2-2-strict_interface_arp/Documentation/filesystems/proc.txt   Mon Sep 10 12:33:30 2001
@@ -1578,6 +1578,14 @@

 Determines whether to send ICMP redirects to other hosts.

+strict_interface_arp
+--------------------
+
+Bind arp request and response to the interface (i.e. an interface uses as source
+and responds to ARPs for an IP address only if the address is configured on the
+interface).
+
+
 Routing settings
 ----------------

diff -Naur linux-2.4.2-2/include/linux/inetdevice.h linux-2.4.2-2-strict_interface_arp/include/linux/inetdevice.h
--- linux-2.4.2-2/include/linux/inetdevice.h  Sat Sep  8 22:08:40 2001
+++ linux-2.4.2-2-strict_interface_arp/include/linux/inetdevice.h      Mon Sep 10 12:33:30 2001
@@ -17,6 +17,7 @@
     int  forwarding;
     int  mc_forwarding;
     int  tag;
+    int  strict_interface_arp;
     void *sysctl;
 };

@@ -47,6 +48,7 @@
 #define IN_DEV_TX_REDIRECTS(in_dev)     (ipv4_devconf.send_redirects || (in_dev)->cnf.send_redirects)
 #define IN_DEV_SEC_REDIRECTS(in_dev)    (ipv4_devconf.secure_redirects || (in_dev)->cnf.secure_redirects)
 #define IN_DEV_IDTAG(in_dev)      ((in_dev)->cnf.tag)
+#define IN_DEV_STRICT_INTF_ARP(in_dev)  (ipv4_devconf.strict_interface_arp || (in_dev)->cnf.strict_interface_arp)

 #define IN_DEV_RX_REDIRECTS(in_dev) \
     ((IN_DEV_FORWARD(in_dev) && \
diff -Naur linux-2.4.2-2/include/linux/sysctl.h linux-2.4.2-2-strict_interface_arp/include/linux/sysctl.h
--- linux-2.4.2-2/include/linux/sysctl.h      Sat Sep  8 22:08:40 2001
+++ linux-2.4.2-2-strict_interface_arp/include/linux/sysctl.h     Mon Sep 10 12:33:30 2001
@@ -327,7 +327,8 @@
     NET_IPV4_CONF_ACCEPT_SOURCE_ROUTE=9,
     NET_IPV4_CONF_BOOTP_RELAY=10,
     NET_IPV4_CONF_LOG_MARTIANS=11,
-    NET_IPV4_CONF_TAG=12
+    NET_IPV4_CONF_TAG=12,
+    NET_IPV4_CONF_STRICT_INTERFACE_ARP=13
 };

 /* /proc/sys/net/ipv6 */
diff -Naur linux-2.4.2-2/net/ipv4/arp.c linux-2.4.2-2-strict_interface_arp/net/ipv4/arp.c
--- linux-2.4.2-2/net/ipv4/arp.c   Sat Sep  8 22:08:40 2001
+++ linux-2.4.2-2-strict_interface_arp/net/ipv4/arp.c   Mon Sep 10 12:33:30 2001
@@ -314,16 +314,43 @@

 static void arp_solicit(struct neighbour *neigh, struct sk_buff *skb)
 {
+    u32 sip;
+    int onlink = 0;
+    struct in_device *in_dev = NULL;
+
     u32 saddr;
     u8  *dst_ha = NULL;
     struct net_device *dev = neigh->dev;
     u32 target = *(u32*)neigh->primary_key;
     int probes = atomic_read(&neigh->probes);

-    if (skb && inet_addr_type(skb->nh.iph->saddr) == RTN_LOCAL)
-         saddr = skb->nh.iph->saddr;
-    else
+    /* if strict_interface_arp then bind arp request saddr to the interface. i.e. saddr must
+     * be on dev where arp request is sent - pflau 06-29-2001 */
+    if (skb) {
+            sip = skb->nh.iph->saddr;
+         in_dev = in_dev_get(dev);
+
+                if (IN_DEV_STRICT_INTF_ARP(in_dev)) {
+              read_lock(&in_dev->lock);
+              for_ifa(in_dev) {
+                   if ((onlink = !(sip ^ ifa->ifa_local)) != 0)
+                                  break;
+              } endfor_ifa(in_dev);
+              read_unlock(&in_dev->lock);
+         } else if (inet_addr_type(skb->nh.iph->saddr) == RTN_LOCAL)
+              onlink = 1;
+
+         if (in_dev)
+              in_dev_put(in_dev);
+    }
+
+
+    if (onlink) {
+         saddr = skb->nh.iph->saddr;
+    } else {
          saddr = inet_select_addr(dev, target, RT_SCOPE_LINK);
+    }
+

     if ((probes -= neigh->parms->ucast_probes) < 0) {
          if (!(neigh->nud_state&NUD_VALID))
@@ -571,6 +598,7 @@
     int addr_type;
     struct in_device *in_dev = in_dev_get(dev);
     struct neighbour *n;
+    int onlink;

 /*
  *  The hardware length of the packet should match the hardware length
@@ -725,8 +753,21 @@
     /* Special case: IPv4 duplicate address detection packet (RFC2131) */
     if (sip == 0) {
          if (arp->ar_op == __constant_htons(ARPOP_REQUEST) &&
-             inet_addr_type(tip) == RTN_LOCAL)
-              arp_send(ARPOP_REPLY,ETH_P_ARP,tip,dev,tip,sha,dev->dev_addr,dev->dev_addr);
+             inet_addr_type(tip) == RTN_LOCAL) {
+              onlink = 0;
+              if (IN_DEV_STRICT_INTF_ARP(in_dev)) {
+                   read_lock(&in_dev->lock);
+                   for_ifa(in_dev) {
+                        if ((onlink = !(tip ^ ifa->ifa_local)) != 0)
+                             break;
+                   } endfor_ifa(in_dev);
+                   read_unlock(&in_dev->lock);
+              } else
+                   onlink = 1;
+
+              if (onlink)
+                   arp_send(ARPOP_REPLY,ETH_P_ARP,tip,dev,tip,sha,dev->dev_addr,dev->dev_addr);
+         }
          goto out;
     }

@@ -739,7 +780,22 @@
          if (addr_type == RTN_LOCAL) {
               n = neigh_event_ns(&arp_tbl, sha, &sip, dev);
               if (n) {
-                   arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);
+                   /* if strict_interface_arp then bind arp response to the interface. */
+                   /* reply only if tip is on dev where arp request is received - 2/5/2001 pflau */
+                   onlink = 0;
+                   if (IN_DEV_STRICT_INTF_ARP(in_dev)) {
+                        read_lock(&in_dev->lock);
+                        for_ifa(in_dev) {
+                             if ((onlink = !(tip ^ ifa->ifa_local)) != 0)
+                                  break;
+                        } endfor_ifa(in_dev);
+                        read_unlock(&in_dev->lock);
+                   } else
+                        onlink = 1;
+
+                   if (onlink) {
+                        arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);
+                   }
                    neigh_release(n);
               }
               goto out;
diff -Naur linux-2.4.2-2/net/ipv4/devinet.c linux-2.4.2-2-strict_interface_arp/net/ipv4/devinet.c
--- linux-2.4.2-2/net/ipv4/devinet.c     Sat Sep  8 22:08:40 2001
+++ linux-2.4.2-2-strict_interface_arp/net/ipv4/devinet.c    Mon Sep 10 12:33:30 2001
@@ -737,6 +737,7 @@
          read_lock(&in_dev->lock);
          for_primary_ifa(in_dev) {
               if (ifa->ifa_scope != RT_SCOPE_LINK &&
+                  !IN_DEV_STRICT_INTF_ARP(in_dev) &&
                   ifa->ifa_scope <= scope) {
                    read_unlock(&in_dev->lock);
                    read_unlock(&inetdev_lock);
@@ -1016,7 +1017,7 @@
 static struct devinet_sysctl_table
 {
     struct ctl_table_header *sysctl_header;
-    ctl_table devinet_vars[13];
+    ctl_table devinet_vars[14];
     ctl_table devinet_dev[2];
     ctl_table devinet_conf_dir[2];
     ctl_table devinet_proto_dir[2];
@@ -1058,6 +1059,9 @@
          &proc_dointvec},
     {NET_IPV4_CONF_TAG, "tag",
      &ipv4_devconf.tag, sizeof(int), 0644, NULL,
+     &proc_dointvec},
+    {NET_IPV4_CONF_TAG, "strict_interface_arp",
+     &ipv4_devconf.strict_interface_arp, sizeof(int), 0644, NULL,
      &proc_dointvec},
      {0}},