Odd result and underlying mistake

Odd result and underlying mistake

[Leroy Tennison]

Posting for other's benefit in case someone else does this.  I searched the web without finding an answer then discovered the issue.  What I saw in a tcpdump output (because things weren't working) was

Request who-has <target IP address> tell <target IP address>

Where <target IP address> was a local interface address, quite odd since the local interface should know its own MAC address.  The problem was that I had accidentally used the local interface IP address in 'ip route add default via <local interface IP address> dev <local interface>' instead of 'ip route add default via <gateway IP address accessible from local interface> dev <interface>'.

Re: Odd result and underlying mistake

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 2973 bytes --]

On 03/16/2018 02:54 PM, Leroy Tennison wrote:
> Posting for other's benefit in case someone else does this.  I searched 
> the web without finding an answer then discovered the issue.  What I 
> saw in a tcpdump output (because things weren't working) was
> 
> Request who-has <target IP address> tell <target IP address>
> 
> Where <target IP address> was a local interface address, quite odd since 
> the local interface should know its own MAC address.

That sounds like a Gratuitous ARP.

> The problem was that I had accidentally used the local interface IP 
> address in 'ip route add default via <local interface IP address> dev 
> <local interface>' instead of 'ip route add default via <gateway IP 
> address accessible from local interface> dev <interface>'.

I think I just reproduced this in a network namespace.

When I do this, I don't see "Request who-has <target IP address> tell 
<target IP address>".  Instead I see "Request who-has <target IP 
address> tell <NetNS IP address>".

19:26:13.919415 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
19:26:14.943348 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
19:26:15.967318 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
19:26:16.991390 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
19:26:18.015337 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28

This in and of itself seems odd to me.  Why is Linux ARPing for an 
address that is obviously not local to the subnet?  (I bound 
192.0.2.1/24, Test-Net-1, to the interface in the NetNS.)

Once I bound 8.8.8.8/32 to the vEth in my main NetNS [1] I saw an ARP 
reply.  But pings to 8.8.8.8/32 timed out.

19:28:22.651995 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
19:28:22.652010 ARP, Reply 8.8.8.8 is-at ca:b0:eb:fa:ef:ab, length 28
19:28:22.652013 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, seq 
1, length 64
19:28:23.711384 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, seq 
2, length 64
19:28:24.735382 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, seq 
3, length 64
19:28:25.759387 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, seq 
4, length 64

When I checked routing on my main NetNS, I found that 192.0.2.0/24 was 
going out my default gateway.  [2]

So I added a route for 192.0.2.1/32 to go out the vEth device that had 
8.8.8.8/32 bound to it.  (But now "via <IP>", just "dev <device>".

ip route add 192.0.2.0/24 dev n1

After doing that, I'm actually able to ping 8.8.8.8 from within the 
network namespace.  IMHO this shouldn't be possible as it's only got a 
route to 192.0.2.0/24.

1) I actually don't know what the main / default routing namespace 
equivalent is.  As far as I can tell, there's no term for it.  At least 
not that I've found.
2) What's surprising by this is that I frequently have 192.0.2.0/24 
bound to a dummy interface on my machine.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3982 bytes --]

Re: Odd result and underlying mistake

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

On 03/16/2018 07:36 PM, Grant Taylor wrote:
> IMHO this shouldn't be possible as it's only got a route to 192.0.2.0/24.

Okay.  Apparently Linux is being smarter than I expect it to be.

If I add a default via a non-local IP, I see ARP requests for said 
non-local IP.  Which will never reply as the IP doesn't exist on the 
vEth pair.

But, that does tell me that Linux would ARP for the gateway IP if it 
didn't already have it.

I just manually added a bogus static ARP entry for the bogus IP and now 
I see Ethernet frames destined to said bogus gateway IP & MAC.

19:40:45.041011 52:29:79:a2:44:b8 > d4:be:d9:8d:80:d3, ethertype IPv4 
(0x0800), length 98: 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11407, 
seq 1, length 64
19:40:46.047385 52:29:79:a2:44:b8 > d4:be:d9:8d:80:d3, ethertype IPv4 
(0x0800), length 98: 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11407, 
seq 2, length 64
19:40:47.071388 52:29:79:a2:44:b8 > d4:be:d9:8d:80:d3, ethertype IPv4 
(0x0800), length 98: 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11407, 
seq 3, length 64
19:40:48.095350 52:29:79:a2:44:b8 > d4:be:d9:8d:80:d3, ethertype IPv4 
(0x0800), length 98: 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11407, 
seq 4, length 64

So … what this tells me is that Linux will use itself as a default 
gateway, and send Ethernet frames out the interface that it's local IP 
is bound to.

First it will ARP for the target IP address.  If it gets a reply, it 
will start sending the traffic.

This can be leveraged by enabling Proxy ARP on the upstream interface.  }:-)



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3982 bytes --]

Re: Odd result and underlying mistake

[cronolog+lartc]

On 2018-03-17 01:36, Grant Taylor wrote:
> On 03/16/2018 02:54 PM, Leroy Tennison wrote:
>> Posting for other's benefit in case someone else does this.  I 
>> searched the web without finding an answer then discovered the 
>> issue.  What I saw in a tcpdump output (because things weren't 
>> working) was
>>
>> Request who-has <target IP address> tell <target IP address>
>>
>> Where <target IP address> was a local interface address, quite odd 
>> since the local interface should know its own MAC address.
>
> That sounds like a Gratuitous ARP.
I agree this is Gratuitous ARP generating this.  It's used to do things 
like IP address conflict detection, and flushing stale ARP caches on 
link-local neighbours, and is quite normal to see.

>
>> The problem was that I had accidentally used the local interface IP 
>> address in 'ip route add default via <local interface IP address> dev 
>> <local interface>' instead of 'ip route add default via <gateway IP 
>> address accessible from local interface> dev <interface>'.
>
> I think I just reproduced this in a network namespace.
>
> When I do this, I don't see "Request who-has <target IP address> tell 
> <target IP address>".  Instead I see "Request who-has <target IP 
> address> tell <NetNS IP address>".
>
> 19:26:13.919415 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
> 19:26:14.943348 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
> 19:26:15.967318 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
> 19:26:16.991390 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
> 19:26:18.015337 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
>
> This in and of itself seems odd to me.  Why is Linux ARPing for an 
> address that is obviously not local to the subnet?  (I bound 
> 192.0.2.1/24, Test-Net-1, to the interface in the NetNS.)
Because next-hop has been set to itself, or more specifically, which 
source interface to use for next-hop with invalid next-hop.  So Linux 
will ARP for anything going out via that source interface's link as if 
it is local connected, and expects a Proxy ARP-enabled device to route 
the packet to the correct destination.  Cisco routers generally have 
Proxy ARP enabled by default, you can also enable it on a Linux router with:

echo 1 > /proc/sys/net/ipv4/conf/{interface}/proxy_arp


>
> Once I bound 8.8.8.8/32 to the vEth in my main NetNS [1] I saw an ARP 
> reply.  But pings to 8.8.8.8/32 timed out.
>
> 19:28:22.651995 ARP, Request who-has 8.8.8.8 tell 192.0.2.1, length 28
> 19:28:22.652010 ARP, Reply 8.8.8.8 is-at ca:b0:eb:fa:ef:ab, length 28
> 19:28:22.652013 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, 
> seq 1, length 64
> 19:28:23.711384 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, 
> seq 2, length 64
> 19:28:24.735382 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, 
> seq 3, length 64
> 19:28:25.759387 IP 192.0.2.1 > 8.8.8.8: ICMP echo request, id 11202, 
> seq 4, length 64
Probably the return route for the ping reply was missing or incorrect in 
the main netNS at this point hence no reply seen, though from below it 
seems you managed to work this bit out.

>
> When I checked routing on my main NetNS, I found that 192.0.2.0/24 was 
> going out my default gateway.  [2]
>
> So I added a route for 192.0.2.1/32 to go out the vEth device that had 
> 8.8.8.8/32 bound to it.  (But now "via <IP>", just "dev <device>".
>
> ip route add 192.0.2.0/24 dev n1
>
> After doing that, I'm actually able to ping 8.8.8.8 from within the 
> network namespace.  IMHO this shouldn't be possible as it's only got a 
> route to 192.0.2.0/24.
As previously, because you still have a default route (but set to a 
link), it will attempt to route via that link.  Proxy ARP in the main 
netNS would normally take care of sorting out layer-2, but since you 
bound 8.8.8.8/32 directly to the main netNS interface, the main netNS 
could reply to the ARP request without Proxy ARP enabled.  Then once you 
added the return path with the above method, a similar process occurs 
for the reply packet, hence completing the loop and allowing you to ping 
and get a reply.

Note that you're only pinging your local 8.8.8.8 from the other netNS, 
not the real 8.8.8.8 on the Internet.  To get a better understanding of 
Proxy ARP, try doing this without binding 8.8.8.8, and enable Proxy ARP, 
routing, and NAT in the main netNS.  You should find with the 3 of those 
things together, you get full Internet access from the other netNS even 
though it doesn't have a proper default gateway address set.

>
> 1) I actually don't know what the main / default routing namespace 
> equivalent is.  As far as I can tell, there's no term for it.  At 
> least not that I've found.
> 2) What's surprising by this is that I frequently have 192.0.2.0/24 
> bound to a dummy interface on my machine 
1) I'm not actually sure either, I've seen it commonly referred to as 
the default namespace though
2) When you bound 8.8.8.8 to the interface, did you add it as an 
additional IP address or did it replace the 192.0.2.0/24 IP address? The 
way you bound 8.8.8.8 would affect the behaviour.  Or if this was a 
different interface, was the dummy interface with 192.0.2.0/24 up or 
down during testing?  Again, the interface state can affect routing 
behaviour.

Re: Odd result and underlying mistake

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 5607 bytes --]

On 03/19/2018 08:29 PM, cronolog+lartc wrote:
> I agree this is Gratuitous ARP generating this.  It's used to do things 
> like IP address conflict detection, and flushing stale ARP caches on 
> link-local neighbours, and is quite normal to see.

ACK

Aside:  I recently ran into a (misconfigured) SDN infrastructure that 
had disabled learning via GARPs.  *facePlant*

> Because next-hop has been set to itself, or more specifically, which 
> source interface to use for next-hop with invalid next-hop.

I question what makes "itself" be an "invalid next-hop".  I also see how 
it's not the "next hop" as in the next device in the line to send 
traffic to.

I suspect this gets into semantics and what was unexpected behavior to me.

I sort of get that the local IP turns into identifying the egress 
interface.  Sort of overloading "next hop" to specify local interface.

> So Linux will ARP for anything going out via that source interface's link 
> as if it is local connected,

This is completely unexpected behavior to me.

I would have expected Linux to return no-route or something like that. 
ARPing for an IP that was not in any local (sub)networks is a total 
surprise to me.  -  Now that I know that's what happens....

> and expects a Proxy ARP-enabled device to route the packet to the correct 
> destination.

Again, completely unexpected behavior.

Aside:  I wonder if this type of behavior was prototypical of devices of 
the era that used Proxy ARP.  Behavior that I've never been exposed to 
(I started networking about 20 years ago, after what I expect was Proxy 
ARP's heyday.)

> Cisco routers generally have Proxy ARP enabled by default,

Um … none of the Cisco gear that I've worked on have had Proxy ARP 
enabled by default.  At least not any that I'm aware of.

I would not be at all surprised to hear that it used to be enabled by 
default.

> you can also enable it on a Linux router with:
> 
> echo 1 > /proc/sys/net/ipv4/conf/{interface}/proxy_arp

*nod*

I've dabbled with Proxy ARP.  I've not had any serious need for it and 
have always used BROUTERs (EBTables) when I needed something like that. 
(Usually for an unrouted protocol.)

> Probably the return route for the ping reply was missing or incorrect in 
> the main netNS at this point hence no reply seen, though from below it 
> seems you managed to work this bit out.

ACK

> As previously, because you still have a default route (but set to a 
> link), it will attempt to route via that link.  Proxy ARP in the main 
> netNS would normally take care of sorting out layer-2, but since you 
> bound 8.8.8.8/32 directly to the main netNS interface, the main netNS 
> could reply to the ARP request without Proxy ARP enabled.  Then once you 
> added the return path with the above method, a similar process occurs 
> for the reply packet, hence completing the loop and allowing you to ping 
> and get a reply.

ACK

I agree that Proxy ARP in the main (default?) NetNS could have handled 
this.  -  I think assuming it is enabled by default is incorrect.

> Note that you're only pinging your local 8.8.8.8 from the other netNS, 
> not the real 8.8.8.8 on the Internet.

Yep.  I'm /acutely/ aware of that.  -  I do lots of things like that for 
PoC and lab testing in my own isolated environments.

> To get a better understanding of Proxy ARP, try doing this without 
> binding 8.8.8.8, and enable Proxy ARP, routing, and NAT in the main netNS.

I'm familiar with what Proxy ARP would have done for the situation.

I'm still surprised at Linux's apparent behavior assuming that something 
like Proxy ARP is in place.

Completely unexpected.

> You should find with the 3 of those things together, you get full Internet 
> access from the other netNS even though it doesn't have a proper default 
> gateway address set.

I get why that would be the case.  -  Suffice it to say that there are 
other things in my network that would prevent it from working.  (I'm 
about 99% certain of that.)

> 1) I'm not actually sure either, I've seen it commonly referred to as 
> the default namespace though

Ya.

I'm also not sure how to return an interface to the main (default) NetNS 
after moving it to another NetNS.  -  Usually I'm dealing with vEth 
pairs that go away when the NetNS they are connected to is deleted.

I also think there is a way that you can move an interface to the same 
NetNS that a different interface is in.  So as long as there was /an/ 
interface in the main (default) NetNS, there is a way.  I'm just not hip 
on the commands / syntax to do so.

> 2) When you bound 8.8.8.8 to the interface, did you add it as an 
> additional IP address or did it replace the 192.0.2.0/24 IP address?

ip addr /add/ 8.8.8.8/32 …

;-)

> The way you bound 8.8.8.8 would affect the behaviour.

For giggles, how would the behavior have been different if I had 
replaced the 192.0.2.0/24 address?

I would have expected the ARPing for 8.8.8.8 to be the same.  -  Unless 
the /32 netmask would have complicated things.  (Sorry, it's been a long 
day.)

> Or if this was a different interface, was the dummy interface with 
> 192.0.2.0/24 up or down during testing?  Again, the interface state can 
> affect routing behaviour.

It was a vEth pair.  I added 8.8.8.8/32 to the interface.  (See the 
above "ip addr" command.

Yep, interfaces being down takes link routes and routes that use 
something on that link out of contention.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3982 bytes --]