Failover route

Failover route

[Leroy Tennison]

SSBoYXZlIGEgZGV2aWNlIHdpdGggdHdvIHJvdXRlcyB0byB0aGUgSW50ZXJuZXQgY29uZmlndXJl
ZCBhcw0KDQpkZWZhdWx0IG5leHRob3AgdmlhIDxJUCBhZGRyZXNzIDE+IGRldiA8TklDIDE+IHdl
aWdodCAxDQpkZWZhdWx0IG5leHRob3AgdmlhIDxJUCBhZGRyZXNzIDI+IGRldiA8TklDIDI+IHdl
aWdodCAxDQoNCklmIEknbSB1bmRlcnN0YW5kaW5nIHdoYXQgSSBzYXcgY29ycmVjdGx5LCBpZiAo
d2hlbikgb25lIGludGVyZmFjZSBmYWlscyBpdCBzdGlsbCB0cmllcyBpdCBldmVyeSBvdGhlciB0
aW1lLiAgV2hhdCBJIHNhdyB3YXMgYW4gb3V0Ym91bmQgcGluZyByZWd1bGFybHkgZmFpbGluZyB0
aGVuIHN1Y2NlZWRpbmcgKGEgZmV3IHN1Y2Nlc3NlcyB0aGVuIGEgZmV3IGZhaWx1cmVzIHRoZW4g
dGhlIHN1Y2Nlc3MvZmFpbHVyZSBwYXR0ZXJuIHJlcGVhdGVkKS4gIEkgdGhlbiBub3RpY2VkIHRo
YXQgb25lIGludGVyZmFjZSB3YXNuJ3QgZnVsbHkgcGx1Z2dlZCBpbiBhbmQsIGFmdGVyIHJlY3Rp
ZnlpbmcgdGhhdCwgcmVjZWl2ZWQgMTAwJSBwaW5nIHJlc3BvbnNlLg0KQWRqdXN0aW5nIHdlaWdo
dHMgaW4gdGhpcyBzaXR1YXRpb24gb25seSBtYWtlcyB0aGluZ3MgYmV0dGVyL3dvcnNlIGRlcGVu
ZGluZyBvbiB3aGljaCBpbnRlcmZhY2UgZmFpbHMsIGlzIHRoZXJlIGEgd2F5IHRvIGNvbmZpZ3Vy
ZSBmYWlsb3ZlciBzbyB0aGF0IG9uZSBpbnRlcmZhY2UgaXMgYWx3YXlzIHRyaWVkIGFuZCB0aGUg
c2Vjb25kIHVzZWQgb25seSBpZiB0aGUgZmlyc3QgZmFpbHM/ICBCb25kaW5nIGlzbid0IGFuIG9w
dGlvbiBiZWNhdXNlIHRoZXNlIGFyZSB0d28gZGlzdGluY3QgcG9pbnQtdG8tcG9pbnQgbGlua3Mu
DQoNCg0KTGVyb3kgVGVubmlzb24NCk5ldHdvcmsgSW5mb3JtYXRpb24vQ3liZXIgU2VjdXJpdHkg
U3BlY2lhbGlzdA0KRTogbGVyb3lAZGF0YXZvaWNlaW50LmNvbQ0KMjIyMCBCdXNoIERyDQpNY0tp
bm5leSwgVGV4YXMNCjc1MDcwDQp3d3cuZGF0YXZvaWNlaW50LmNvbQ0KVGhpcyBtZXNzYWdlIGhh
cyBiZWVuIHNlbnQgb24gYmVoYWxmDQpvZiBhIGNvbXBhbnkgdGhhdCBpcyBwYXJ0IG9mIHRoZSBI
YXJyaXMgT3BlcmF0aW5nIEdyb3VwIG9mDQpDb25zdGVsbGF0aW9uIFNvZnR3YXJlIEluYy4gVGhl
c2UgY29tcGFuaWVzIGFyZSBsaXN0ZWQNCmhlcmUNCi4NCklmIHlvdSBwcmVmZXIgbm90IHRvIGJl
IGNvbnRhY3RlZCBieSBIYXJyaXMNCk9wZXJhdGluZyBHcm91cA0KcGxlYXNlIG5vdGlmeSB1cw0K
Lg0KVGhpcyBtZXNzYWdlIGlzIGludGVuZGVkIGV4Y2x1c2l2ZWx5IGZvciB0aGUNCmluZGl2aWR1
YWwgb3IgZW50aXR5IHRvIHdoaWNoIGl0IGlzIGFkZHJlc3NlZC4gVGhpcyBjb21tdW5pY2F0aW9u
DQptYXkgY29udGFpbiBpbmZvcm1hdGlvbiB0aGF0IGlzIHByb3ByaWV0YXJ5LCBwcml2aWxlZ2Vk
IG9yDQpjb25maWRlbnRpYWwgb3Igb3RoZXJ3aXNlIGxlZ2FsbHkgZXhlbXB0IGZyb20gZGlzY2xv
c3VyZS4gSWYgeW91IGFyZQ0Kbm90IHRoZSBuYW1lZCBhZGRyZXNzZWUsIHlvdSBhcmUgbm90IGF1
dGhvcml6ZWQgdG8gcmVhZCwgcHJpbnQsDQpyZXRhaW4sIGNvcHkgb3IgZGlzc2VtaW5hdGUgdGhp
cyBtZXNzYWdlIG9yIGFueSBwYXJ0IG9mIGl0LiBJZiB5b3UNCmhhdmUgcmVjZWl2ZWQgdGhpcyBt
ZXNzYWdlIGluIGVycm9yLCBwbGVhc2Ugbm90aWZ5IHRoZSBzZW5kZXINCmltbWVkaWF0ZWx5IGJ5
IGUtbWFpbCBhbmQgZGVsZXRlIGFsbCBjb3BpZXMgb2YgdGhlDQptZXNzYWdlLg0KDQo

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 3604 bytes --]

On 3/12/19 8:32 AM, Leroy Tennison wrote:
> I have a device with two routes to the Internet configured as
> 
> default nexthop via <IP address 1> dev <NIC 1> weight 1
> default nexthop via <IP address 2> dev <NIC 2> weight 1
>  > If I'm understanding what I saw correctly, if (when) one interface fails
> it still tries it every other time.  What I saw was an outbound ping 
> regularly failing then succeeding (a few successes then a few failures 
> then the success/failure pattern repeated).  I then noticed that one 
> interface wasn't fully plugged in and, after rectifying that, received 
> 100% ping response.
> 
> Adjusting weights in this situation only makes things better/worse 
> depending on which interface fails, is there a way to configure failover 
> so that one interface is always tried and the second used only if the 
> first fails?  Bonding isn't an option because these are two distinct 
> point-to-point links.

You're stepping into a very murky area.  I've not been able to get what 
I think you're wanting to do to work reliably after multiple attempts in 
20<ish> years.  -  Hopefully someone knows something that I don't.

I have gotten Dead Gateway Detection to work.  But that only removes a 
route when the Dead Gateway is detected and will not detect when the 
gateway is alive, nor will it re-add the gateway.

I have had issues with DGD not working (without fobbing some knobs) if 
link is not lost.  So an SDSL issue on the other side of a bridging 
modem isn't detected properly.

I have not needed to fight this battle in a while.  But I've come to the 
conclusion that I'd do it in multiple stages.  I'd likely use a NetNS 
(or possibly heavy weight VM) to function as a local gateway that would 
monitor the immediate upstream gateway.  It would then conditionally 
advertise itself as a default gateway to a (main) downstream routing 
instance that would do ECMP across the viable upstream routing instances.

You also have the problem of how to detect and respond to problems past 
the ISP's default gateway.  I.e. if the problem is in their core or 
their upstream connections.  In this scenario, all the logic testing 
reachability to the ISP is largely useless, if not actually providing a 
false signal.  At least that (or something resembling it) had been my plan.

One of the complications with doing this is how do you have routes to 
probe things without having the kernel use said routes for data traffic. 
  This starts to quickly get into Policy Based Routing and / or 
containerization with separate routes.  I suppose you can also have a 
route to the probe destination that is more specific than the default. 
Now we're down the rabbit hole.

Recently I've been wondering about doing something to monitor the 
traffic and actually watch for a sufficient imbalance in outgoing SYN vs 
incoming SYN+ACK.  If that gets too far towards SYN only, consider the 
route as being problematic.  This would account for the ISP upstream 
issue too.  Thus far I've only been thinking about the route in it's 
entirety.  But it should be possible to also be more granular.

I have long wanted Bidirectional Forwarding Detection to be a thing 
between my router and my ISPs.  Unfortunately, (it's my understanding 
that) BFD requires active support from the other side.

I would love to learn that someone has solved this with just the Linux 
kernel and not requiring some sort of daemon, be it a proper routing 
daemon or something like I've described above.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

On 3/12/19 9:36 AM, Grant Taylor wrote:
> Recently I've been wondering about doing something to monitor the 
> traffic and actually watch for a sufficient imbalance in outgoing SYN vs 
> incoming SYN+ACK.  If that gets too far towards SYN only, consider the 
> route as being problematic.  This would account for the ISP upstream 
> issue too.  Thus far I've only been thinking about the route in it's 
> entirety.  But it should be possible to also be more granular.

I think it would be possible to do similar for ICMP Echo (Request) (Type 
8) and Echo Reply (Type 0).

It may be possible to use Connection Tracking's state transition 
mechanisms too.  This might mean it's possible to use UDP-Lite protocol 
connection tracking support for UDP.

I've pontificated packet captures (possibly via libcap?) as well as 
selective IPTables rules to queue (copies of) packets to user space, 
possibly via the NetLink interface.

One of the other problems that I have (mentally) run into is how to 
properly handle the routing between multiple instances.  Do you 
advertise an unreachable route, thus causing the other instances to be 
chosen?  Do you blindly have the upstream instance re-route across to 
other upstream instances?

How do you have the main instance and the upstream instances communicate 
with each other?  (This seems like the domain of a routing protocol.)

Do we need to periodically re-try traffic to problematic IPs and update 
routes accordingly?

I would love to learn that there are solutions to these problems and 
what they are.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Erik Auerswald]

Hi,

On 3/12/19 15:32, Leroy Tennison wrote:
> I have a device with two routes to the Internet configured as
> 
> default nexthop via <IP address 1> dev <NIC 1> weight 1
> default nexthop via <IP address 2> dev <NIC 2> weight 1
> 
> If I'm understanding what I saw correctly, if (when) one interface fails it still tries it every other time.  What I saw was an outbound ping regularly failing then succeeding (a few successes then a few failures then the success/failure pattern repeated).  I then noticed that one interface wasn't fully plugged in and, after rectifying that, received 100% ping response.
> Adjusting weights in this situation only makes things better/worse depending on which interface fails, is there a way to configure failover so that one interface is always tried and the second used only if the first fails?  Bonding isn't an option because these are two distinct point-to-point links.

As a crude method I have used a cron job to ping the gateways and
add or remove (default) routes based on ping results in the past.
You could use iproute2's 'ip link' to check interface status instead
(or first).

You may be able to subscribe to Netlink messages to get informed
about interface state changes. At least for a prototype it might
suffice to use iproute2's 'ip monitor' to receive Netlink messages,
but I have not yet tried this. Routing daemons are supposed to do
that, so using one with static default routes might give you the
intended results.

Thanks,
Erik

Re: Failover route

[Erik Auerswald]

Hi,

On 3/12/19 16:36, Grant Taylor wrote:
> On 3/12/19 8:32 AM, Leroy Tennison wrote:
>> I have a device with two routes to the Internet configured as
>>
>> default nexthop via <IP address 1> dev <NIC 1> weight 1
>> default nexthop via <IP address 2> dev <NIC 2> weight 1
>>  > If I'm understanding what I saw correctly, if (when) one interface 
>> fails
>> it still tries it every other time.  What I saw was an outbound ping 
>> regularly failing then succeeding (a few successes then a few failures 
>> then the success/failure pattern repeated).  I then noticed that one 
>> interface wasn't fully plugged in and, after rectifying that, received 
>> 100% ping response.
>> [...]
> I have long wanted Bidirectional Forwarding Detection to be a thing 
> between my router and my ISPs.  Unfortunately, (it's my understanding 
> that) BFD requires active support from the other side.

BFD echo mode does not require support from the other side. It works by
sending an IP packet destined to the sending interface out an interface.
The upstream side is supposed to send this packet back through the same
interface (this is IP, not Ethernet).

I have not yet tried this on Linux, but BFD echo modes is used for short
failure detection times in larger networks, because often the line card
(or port ASIC) can generate and check the packets without CPU
processing. A BFD control session is still usually used between adjacent
devices, but it is not strictly necessary.

Thanks,
Erik

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

On 3/16/19 10:22 AM, Erik Auerswald wrote:
> Hi,

Hi,

> As a crude method I have used a cron job to ping the gateways and add 
> or remove (default) routes based on ping results in the past.  You could 
> use iproute2's 'ip link' to check interface status instead (or first).

Is the following true:

  · The NIC link is staying up.
  · The IP is remaining assigned to the interface.
  · You are conditionally adding / changing / removing the default 
gateway that is the far side of the link.
  · You have some sort of ECMP and / or PBR in place to work with 
multiple default gateways.

I found issues with relying on link status in the following scenario:

+-------+     +------------+     /   +-----+   === Ethernet
| Linux +=====+ SDSL Modem +----/----+ ISP |   --- SDSL
+-------+     +------------+   /     +-----+    /  Break

Linux will think that the link is up, because it is up to the SDSL 
modem.  Yet there is no end-to-end connection between Linux and the ISP 
because there is a break in the SDSL connection to between the SDSL 
modem and the ISP.

I have fought this very scenario.  The same could be true if you use an 
Ethernet switch as a media converter going between copper and fiber.

The problem is that Linux sees the link as always being up, despite 
there being a break further down the line.

> You may be able to subscribe to Netlink messages to get informed about 
> interface state changes. At least for a prototype it might suffice to 
> use iproute2's 'ip monitor' to receive Netlink messages, but I have not 
> yet tried this.

I like the idea and the ingenuity.  Unfortunately it suffers from the 
underlying problem above.

> Routing daemons are supposed to do that, so using one with static default 
> routes might give you the intended results.

I don't know that I've heard that routing daemons are /supposed/ /to/ 
rely on local link state.  I think that many do for various reasons. 
But I've never heard of any such mandate before.

To me, monitoring for loss of a link is mostly an optimization for the 
routing daemon to know that a link is unusable in short order and can 
remove the associated routes.  I'm fairly certain that the Linux kernel 
will do the same when link goes down, and remove routes that were using 
that link.

Sure, there are some link-state protocols that may want to know this 
information as part of the protocol's operation.  As such the routing 
daemon that implements said protocol would want to know too.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

On 3/16/19 10:33 AM, Erik Auerswald wrote:
> Hi,

Hi,

> BFD echo mode does not require support from the other side.

Full Stop!  What‽

…reading hiatus…

Very interesting.

> It works by sending an IP packet destined to the sending interface out 
> an interface.  The upstream side is supposed to send this packet back 
> through the same interface (this is IP, not Ethernet).

I guess BFD Echo Mode is really just an IP packet that exercises the 
remote forwarding engine.

It should work on any type of link where you can send a packet addressed 
to yourself (that happens to be from yourself) and sent out a link to a 
device that that should forward traffic destined to your local IP address.

I wonder if µRPF would interfere with this at all.  I doubt it, because 
the source IP would be coming in an interface that is an outgoing route 
to said IP as a destination.

I can see how some additional checking above and beyond µRPF might take 
objection to traffic coming in an interface and immediately going back 
out the same interface.  As in why is that traffic coming in said 
interface in the first place.  But that's not µRPF as I understand it.

> I have not yet tried this on Linux, but BFD echo modes is used for 
> short failure detection times in larger networks, because often the 
> line card (or port ASIC) can generate and check the packets without CPU 
> processing. A BFD control session is still usually used between adjacent 
> devices, but it is not strictly necessary.

I'm very intrigued by the idea of using (what I'm going to call) BFD-EM 
/without/ BFD control sessions.  I think this means that BFD-EM will 
work against any device that will forward packets back to you.

Very interesting.

Thank you for sharing Erik.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Erik Auerswald]

Hi,

On 3/16/19 18:29, Grant Taylor wrote:
> On 3/16/19 10:33 AM, Erik Auerswald wrote:
>> BFD echo mode does not require support from the other side.
> 
> Full Stop!  What‽
> 
> …reading hiatus…
> 
> Very interesting.
> 
>> It works by sending an IP packet destined to the sending interface out 
>> an interface.  The upstream side is supposed to send this packet back 
>> through the same interface (this is IP, not Ethernet).
> 
> I guess BFD Echo Mode is really just an IP packet that exercises the 
> remote forwarding engine.
> 
> It should work on any type of link where you can send a packet addressed 
> to yourself (that happens to be from yourself) and sent out a link to a 
> device that that should forward traffic destined to your local IP address.
> 
> I wonder if µRPF would interfere with this at all.  I doubt it, because 
               ^
               u

This is called uRPF for "unicast RPF" as opposed to the multicast RPF
check this is based on.

> the source IP would be coming in an interface that is an outgoing route 
> to said IP as a destination.

Indeed uRPF should allow that packet, as the source address is found
off the receiving interface (usually via a directly attached network).

> I can see how some additional checking above and beyond µRPF might take 
> objection to traffic coming in an interface and immediately going back 
> out the same interface.  As in why is that traffic coming in said 
> interface in the first place.  But that's not µRPF as I understand it.

That could be done via ACLs, but would be unusual AFAIK. The uRPF
functionality itself should not interfere.

>> I have not yet tried this on Linux, but BFD echo modes is used for 
>> short failure detection times in larger networks, because often the 
>> line card (or port ASIC) can generate and check the packets without 
>> CPU processing. A BFD control session is still usually used between 
>> adjacent devices, but it is not strictly necessary.
> 
> I'm very intrigued by the idea of using (what I'm going to call) BFD-EM 
> /without/ BFD control sessions.  I think this means that BFD-EM will 
> work against any device that will forward packets back to you.
> 
> Very interesting.
> 
> Thank you for sharing Erik.

You're welcome. :)

Thanks,
Erik

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]

On 3/16/19 11:29 AM, Grant Taylor wrote:
> I'm very intrigued by the idea of using (what I'm going to call) BFD-EM 
> /without/ BFD control sessions.  I think this means that BFD-EM will 
> work against any device that will forward packets back to you.

I can see how BFD-EM can work in the SDSL failure scenario that I described.

+-------+     +------------+     /   +-----+   === Ethernet
| Linux +=====+ SDSL Modem +----/----+ ISP |   --- SDSL
+-------+     +------------+   /     +-----+    /  Break

I think the sequence of events would be something like the following:

1)  BFD-EM would send an IP packet (Ethernet frame) from it's Link-Net 
IP & local MAC to it's Link-Net IP and the upstream router's MAC.

2)  The upstream router will receive the Ethernet frame because it's to 
the upstream router's MAC.

3)  The upstream router will then route the IP packet out the proper 
interface for the destination IP of the packet.  This should be the same 
interface that the packet (Ethernet frame) came in on.

4)  BFD-EM will receive the IP packet from it's Link-Net IP & the 
upstream router's MAC to it's Link-Net IP and local MAC.

5)  BFD-EM successfully confirmed that there is end-to-end connectivity 
across the link via steps 1 through 4.

I like this.

It shouldn't require any support from the upstream router other than 
standard routing, which it is already doing.

I can also see how a BFD-EM implementation would only need to 
dynamically modify any route(s) using the upstream router.  It wouldn't 
need to remove or re-add any IPs to the interface.  (No fighting with 
NetworkManager, et al.)

I can see how it would be possible to use established system 
initialization scripts to bring up the interface with an IP and no 
default gateway.  Then have the BFD-EM implementation add / remove the 
default gateway as appropriate in response to the successful completion 
of steps 1 through 4 above.

I now wonder if it would be possible to have a BFD-EM implementation 
that doesn't alter the routing table itself.  Instead have it speak a 
routing protocol to a traditional routing daemon and allow it to manage 
the routing table.  Maybe a simple stateless routing protocol, RIP comes 
to mind.

> Very interesting. 

Very interesting indeed.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On 3/16/19 11:40 AM, Erik Auerswald wrote:
> Hi,

Hi,

> This is called uRPF for "unicast RPF" as opposed to the multicast RPF 
> check this is based on.

Thank you for the correction.  :-)

> Indeed uRPF should allow that packet, as the source address is found 
> off the receiving interface (usually via a directly attached network).

*nod*

> That could be done via ACLs, but would be unusual AFAIK. The uRPF 
> functionality itself should not interfere.

I agree that this would be atypical.

As I type this, I wonder if there might be any problems related to uRPF 
/sending/ the BFD-EM packet.  I think it's going to need to be sent in 
such a way as to bypass the local routing stack.

But that's on a system that I control and I have options to overcome any 
local limitations.  }:-)



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Erik Auerswald]

Hi,

On 3/16/19 18:12, Grant Taylor wrote:
> On 3/16/19 10:22 AM, Erik Auerswald wrote:
> 
>> As a crude method I have used a cron job to ping the gateways and add 
>> or remove (default) routes based on ping results in the past.  You 
>> could use iproute2's 'ip link' to check interface status instead (or 
>> first).
> 
> Is the following true:
> 
>   · The NIC link is staying up.
>   · The IP is remaining assigned to the interface.
>   · You are conditionally adding / changing / removing the default 
> gateway that is the far side of the link.
>   · You have some sort of ECMP and / or PBR in place to work with 
> multiple default gateways.
> 
> I found issues with relying on link status in the following scenario:
> 
> +-------+     +------------+     /   +-----+   == Ethernet
> | Linux +===+ SDSL Modem +----/----+ ISP |   --- SDSL
> +-------+     +------------+   /     +-----+    /  Break
> 
> Linux will think that the link is up, because it is up to the SDSL 
> modem.  Yet there is no end-to-end connection between Linux and the ISP 
> because there is a break in the SDSL connection to between the SDSL 
> modem and the ISP.
> 
> I have fought this very scenario.  The same could be true if you use an 
> Ethernet switch as a media converter going between copper and fiber.
> 
> The problem is that Linux sees the link as always being up, despite 
> there being a break further down the line.

If you have an xDSL modem such that the next-hop IP address is on the
ISP side, using ping to probe reachability of the next-hop should just
work. You could look into using the -r and -I options of ping (from
iputils as packaged by Debian, in my case).

If you want to check to some specific IPs or hosts behind one ISP, you
can use static host routes (that are not withdrawn via any daemon and
stay up even when the interface status goes down) pointing out the
specific interface to check. Of course, those IPs should not provide
other useful service, or you risk blackholing them on connection
failures.

>> You may be able to subscribe to Netlink messages to get informed about 
>> interface state changes. At least for a prototype it might suffice to 
>> use iproute2's 'ip monitor' to receive Netlink messages, but I have 
>> not yet tried this.
> 
> I like the idea and the ingenuity.  Unfortunately it suffers from the 
> underlying problem above.
> 
>> Routing daemons are supposed to do that, so using one with static 
>> default routes might give you the intended results.
> 
> I don't know that I've heard that routing daemons are /supposed/ /to/ 
> rely on local link state.  I think that many do for various reasons. But 
> I've never heard of any such mandate before.

I'd say that is more of a tradition. Routing daemons are often written
to mimic Cisco IOS behaviour, and there static routes towards an
IP that is not reachable (except for a possible default route) are
removed from the RIB and FIB. This is not a requirement for a routing
daemon, and e.g. Cisco IOS allows overriding this default behaviour.

> To me, monitoring for loss of a link is mostly an optimization for the 
> routing daemon to know that a link is unusable in short order and can 
> remove the associated routes.  I'm fairly certain that the Linux kernel 
> will do the same when link goes down, and remove routes that were using 
> that link.

IIRC the Linux kernel does not remove those routes itself.

> Sure, there are some link-state protocols that may want to know this 
> information as part of the protocol's operation.  As such the routing 
> daemon that implements said protocol would want to know too.

Thanks,
Erik

Re: Failover route

[Erik Auerswald]

Hi,

On 3/16/19 18:51, Grant Taylor wrote:
> On 3/16/19 11:40 AM, Erik Auerswald wrote:
> [...]
> As I type this, I wonder if there might be any problems related to uRPF 
> /sending/ the BFD-EM packet.  I think it's going to need to be sent in 
> such a way as to bypass the local routing stack.
> 
> But that's on a system that I control and I have options to overcome any 
> local limitations.  }:-)

Yeah, one needs to send a packet destined to the same machine out an
interface. That might need some trickery.

I remember some related trickery from a Project Zero blog post:
googleprojectzero.blogspot.de/2015/12/fireeye-exploitation-project-zeros.html

A BFD daemon using RAW sockets (neither UDP nor TCP) might not need
something like the above, similar to ping -I <Iface> -r <TargetIP>.

Thanks,
Erik

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 2155 bytes --]

On 3/16/19 11:57 AM, Erik Auerswald wrote:
> Hi,

Hi,

> If you have an xDSL modem such that the next-hop IP address is on the 
> ISP side, using ping to probe reachability of the next-hop should just 
> work.

I want to agree with you.  Though I have disqualified ping (ICMP) as a 
reliable test in the past for a reason that I can't recall at the 
moment.  I do know that I've run into ISPs that filter ICMP or do other 
nasty things with it, rendering it unreliable.

I think that BFD-EM would avoid some of thee ICMP scheenanigans that 
I've seen before.  The /only/ thing that the upstream router needs to do 
is route a packet.  Conversely, the upstream would need to respond ICMP 
itself and not be filtered.

There's also a chance that the router might respond to ICMP but not 
actually forward packets.  Possibly during maintenance.

> You could look into using the -r and -I options of ping (from iputils 
> as packaged by Debian, in my case).

Good to know.

> If you want to check to some specific IPs or hosts behind one ISP, 
> you can use static host routes (that are not withdrawn via any daemon 
> and stay up even when the interface status goes down) pointing out the 
> specific interface to check. Of course, those IPs should not provide 
> other useful service, or you risk blackholing them on connection failures.

Yep.  Hence my previous comments about monitoring SYN / ACK and / or 
ICMP echo request / echo reply associations.

> I'd say that is more of a tradition. Routing daemons are often written 
> to mimic Cisco IOS behaviour, and there static routes towards an IP 
> that is not reachable (except for a possible default route) are removed 
> from the RIB and FIB. This is not a requirement for a routing daemon, 
> and e.g. Cisco IOS allows overriding this default behaviour.

ACK

> IIRC the Linux kernel does not remove those routes itself.

I've had the kernel remove routes when an interface id brought down.  I 
don't recall off hand if it did so on link loss (up / down condition).

> Thanks,

You're welcome.

Thank you.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

On 3/16/19 12:04 PM, Erik Auerswald wrote:
> Hi,

Hi,

> Yeah, one needs to send a packet destined to the same machine out an 
> interface. That might need some trickery.

The more that I think about it, the more that I think:

  · BFD-EM style to-be-routed packet might work better than ICMP.
  · link-net IPs shouldn't be a problem.

> I remember some related trickery from a Project Zero blog post: 
> googleprojectzero.blogspot.de/2015/12/fireeye-exploitation-project-zeros.html

Either I'm not understanding what you're referring to.  Or I don't see 
how essentially sniffing a (mirror / SPAN) port differs from what I said 
previously about sniffing traffic.

Or are you talking about applying sniffing to the BFD-EM frame?

> A BFD daemon using RAW sockets (neither UDP nor TCP) might not need 
> something like the above, similar to ping -I <Iface> -r <TargetIP>.

I'm really starting to question if I'm / we're not over complicating this.

I can't think of a reason why BFD-EM to from & to the local link-net IP 
via the far end router's MAC address won't work.

The link-net IPs should only be used for things on the link.  So I don't 
see any disadvantage of said IPs not being reachable if the link is down 
/ down.  Arguably, that's a state that BFD-EM should account for.  The 
reason for using BFD-EM would want to know about such a down / down state.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Erik Auerswald]

Hi,

On Sun, Mar 17, 2019 at 12:38:39PM -0600, Grant Taylor wrote:
> On 3/16/19 12:04 PM, Erik Auerswald wrote:
> 
> >Yeah, one needs to send a packet destined to the same machine out
> >an interface. That might need some trickery.
> 
> The more that I think about it, the more that I think:
> 
>  · BFD-EM style to-be-routed packet might work better than ICMP.
>  · link-net IPs shouldn't be a problem.
> 
> >I remember some related trickery from a Project Zero blog post: googleprojectzero.blogspot.de/2015/12/fireeye-exploitation-project-zeros.html
> 
> Either I'm not understanding what you're referring to.  Or I don't
> see how essentially sniffing a (mirror / SPAN) port differs from
> what I said previously about sniffing traffic.

I am referring to the following:

--------
In this case, I want the simulated network on 192.168.2.0/24 and the
management interface on 192.168.1.1/24. This requires adjusting the
local routing table and creating device specific rules accordingly.

# First, make sure Linux will accept local<->hub<->local traffic
echo 1 > /proc/sys/net/ipv4/conf/all/accept_local

# Give the two interfaces connected to the hub static addresses.
ip addr add 192.168.2.2 dev eth2
ip addr add 192.168.2.1 dev eth3

# Route traffic destined for each address to the *opposite* device
ip route add 192.168.2.1 dev eth2
ip route add 192.168.2.2 dev eth3

# Force Linux to forget that these are local interfaces so the traffic actually reaches the hub.
ip route del 192.168.2.2 table local
ip route del 192.168.2.1 table local

# Now add corresponding rules for each interface
ip rule add iif eth2 lookup 100
ip route add local 192.168.2.2 dev eth2 table 100
ip rule add iif eth3 lookup 101
ip route add local 192.168.2.1 dev eth3 table 101
--------

> Or are you talking about applying sniffing to the BFD-EM frame?
> 
> >A BFD daemon using RAW sockets (neither UDP nor TCP) might not
> >need something like the above, similar to ping -I <Iface> -r
> ><TargetIP>.
> 
> I'm really starting to question if I'm / we're not over complicating this.
> 
> I can't think of a reason why BFD-EM to from & to the local link-net
> IP via the far end router's MAC address won't work.

Yes, if you craft a packet accordingly. But what does the TCP/IP stack
do if you send a packet to a local IP address? The BFD Echo packet is
addressed to the local IP of the sending interface.

> The link-net IPs should only be used for things on the link.  So I
> don't see any disadvantage of said IPs not being reachable if the
> link is down / down.  Arguably, that's a state that BFD-EM should
> account for.  The reason for using BFD-EM would want to know about
> such a down / down state.

The question is how to create a packet with the local IP and the remote
MAC address, and send it out the local interface.

One way is editing a PCAP file with a hex editor and using tcpreplay to
send the PCAP data as a packet. ;-)

Thanks,
Erik
-- 
In the beginning, there was static routing.
                        -- RFC 1118

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 3637 bytes --]

On 3/18/19 8:53 AM, Erik Auerswald wrote:
> Hi,

Hi,

> I am referring to the following:
> 
> --------
> In this case, I want the simulated network on 192.168.2.0/24 and the 
> management interface on 192.168.1.1/24. This requires adjusting the 
> local routing table and creating device specific rules accordingly.

I'm not following what you're wanting to do, why you're wanting to do 
it, and how it's related to the thread.  —  There's apparently too much 
blood in my caffeine stream to grok this properly.

> # First, make sure Linux will accept local<->hub<->local traffic
> echo 1 > /proc/sys/net/ipv4/conf/all/accept_local

This is new to me.

It looks like accept_local is a knob to disable basic sanity checks.

I can't articulate what the sanity checks are and why it's good that 
they're there in the first place.

#https://i.etsystatic.com/5918013/r/il/48c7e5/289046102/il_794xN.289046102.jpg

> # Give the two interfaces connected to the hub static addresses.
> ip addr add 192.168.2.2 dev eth2
> ip addr add 192.168.2.1 dev eth3

Okay.

(I've not tested to find out.)  What netmask do these IPs get?  /24 
since they are in the class c address range?

> # Route traffic destined for each address to the *opposite* device
> ip route add 192.168.2.1 dev eth2
> ip route add 192.168.2.2 dev eth3

Okay.

These are device routes that get added to the main routing table (ID 
32766).  They state that the destination IPs are available out the devices.

> # Force Linux to forget that these are local interfaces so the traffic actually reaches the hub.
> ip route del 192.168.2.2 table local
> ip route del 192.168.2.1 table local

Removing the (local) routes from the local routing table (ID 0).

> # Now add corresponding rules for each interface
> ip rule add iif eth2 lookup 100
> ip route add local 192.168.2.2 dev eth2 table 100
> ip rule add iif eth3 lookup 101
> ip route add local 192.168.2.1 dev eth3 table 101

(Re)adding the (local) routes that were removed from the local table to 
tables 100 and 101 (respectively) and adding ip rules (PBR) to choose 
the routing tables based on the incoming interface.

If I'm tracking properly, the system will effectively not realize that 
192.168.2.2 and 192.168.2.1 are local because the ip rules won't search 
tables 100 or 101.  But it will have (device) routes to them, thus it 
will be able to send traffic to them.

There is the question of what source IP is used.

> --------
> 
> Yes, if you craft a packet accordingly. But what does the TCP/IP stack 
> do if you send a packet to a local IP address? The BFD Echo packet is 
> addressed to the local IP of the sending interface.

I think, by default, the kernel will try loop internally.

> The question is how to create a packet with the local IP and the remote 
> MAC address, and send it out the local interface.
> 
> One way is editing a PCAP file with a hex editor and using tcpreplay to 
> send the PCAP data as a packet. ;-)

I feel like there are some tools that can be used to create and send 
custom Ethernet frames.

A quick web search renders this: 
http://www.microhowto.info/howto/send_an_arbitrary_ethernet_frame_using_an_af_packet_socket_in_c.html

I feel like there are other tools that usually fall into the gray / 
black hat camps that can also do this.

I also wonder if there are some ways to do this with network namespaces 
to avoid some of the routing complexities / hacks above.

> Thanks,

You're welcome.

Now to go find some much needed caffeine.



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]

Re: Failover route

[Erik Auerswald]

Hi,


On Mon, Mar 18, 2019 at 10:57:59AM -0600, Grant Taylor wrote:
> On 3/18/19 8:53 AM, Erik Auerswald wrote:
> 
> >I am referring to the following:
> >
> >--------
> >[...]
> I'm not following what you're wanting to do, why you're wanting to
> do it, and how it's related to the thread.
> [...]
> ># Force Linux to forget that these are local interfaces so the traffic actually reaches the hub.
> >ip route del 192.168.2.2 table local
> >ip route del 192.168.2.1 table local
> 
> Removing the (local) routes from the local routing table (ID 0).

I was alluding to this part specifically, i.e. tricking the kernel into
sending a packet out an interface although it is addressed to a local
IP address.

> ># Now add corresponding rules for each interface
> >ip rule add iif eth2 lookup 100
> >ip route add local 192.168.2.2 dev eth2 table 100
> >ip rule add iif eth3 lookup 101
> >ip route add local 192.168.2.1 dev eth3 table 101
> 
> (Re)adding the (local) routes that were removed from the local table
> to tables 100 and 101 (respectively) and adding ip rules (PBR) to
> choose the routing tables based on the incoming interface.
> 
> If I'm tracking properly, the system will effectively not realize
> that 192.168.2.2 and 192.168.2.1 are local because the ip rules
> won't search tables 100 or 101.  But it will have (device) routes to
> them, thus it will be able to send traffic to them.

Exactly. In this case two interfaces are used that shall send data to
each other over an external network (a hub), thus source based routing
is used. Source based routing is probably not needed for the BFD-EM idea.

> There is the question of what source IP is used.

In the blog post, the IP of the egress interface was used. As I understand
it, after the kernel determines which interface to send a packet out of,
it determines the "best" IP to use.

I do think that this is quite implementation specific, and I would not
bet that the above behaviour is seen as part of Linux's stable API.

> >--------
> >
> >Yes, if you craft a packet accordingly. But what does the TCP/IP
> >stack do if you send a packet to a local IP address? The BFD Echo
> >packet is addressed to the local IP of the sending interface.
> 
> I think, by default, the kernel will try loop internally.
> 
> >The question is how to create a packet with the local IP and the
> >remote MAC address, and send it out the local interface.
> >
> >One way is editing a PCAP file with a hex editor and using
> >tcpreplay to send the PCAP data as a packet. ;-)
> 
> I feel like there are some tools that can be used to create and send
> custom Ethernet frames.
> 
> A quick web search renders this: http://www.microhowto.info/howto/send_an_arbitrary_ethernet_frame_using_an_af_packet_socket_in_c.html

Yes, I'd look into something like this for a BFD-EM implementation.

> I feel like there are other tools that usually fall into the gray /
> black hat camps that can also do this.

Those could be used for a relatively simple proof-of-concept.

> I also wonder if there are some ways to do this with network
> namespaces to avoid some of the routing complexities / hacks above.

I'd expect that using AF_PACKET, SOCK_RAW removes the need for the hacks
used by Project Zero.  They were using e.g. curl instead of writing
their own network application.

Thanks,
Erik
-- 
That's 10 'programmer minutes', i.e. less than a day.
                        -- Russ Cox

Re: Failover route

[Grant Taylor]

[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]

On 3/19/19 6:26 AM, Erik Auerswald wrote:
> Hi,

Hi,

> I was alluding to this part specifically, i.e. tricking the kernel into 
> sending a packet out an interface although it is addressed to a local 
> IP address.

I agree that's what is being done in that case.

The kicker is that it's destined for an IP that the kernel / routing 
doesn't realize is local based on the modifications to routing tables.

Even that, traffic is from one IP (192.168.2.1 / 192.168.2.2) and going 
to a different IP (192.168.2.2 / 192.168.2.1 respectively).

Sending from an IP to the same IP is decidedly different.

> Exactly. In this case two interfaces are used that shall send data to 
> each other over an external network (a hub), thus source based routing 
> is used. Source based routing is probably not needed for the BFD-EM idea.

I think the PBR would be significantly different, if not more complex 
for BFD-EM.

I'm currently of the mindset that this likely shouldn't be done with 
routing and instead should be done at a lower level.

The reply / incoming packet can probably be processed at a normal level.

> In the blog post, the IP of the egress interface was used. As I understand 
> it, after the kernel determines which interface to send a packet out of, 
> it determines the "best" IP to use.
> 
> I do think that this is quite implementation specific, and I would not 
> bet that the above behaviour is seen as part of Linux's stable API.

I think the behavior represented in the article should be quite stable. 
It's not even an API thing.  It's a config thing.

 From a routing perspective:

  · There is no 192.168.2.* IP in the local or main routing table.  Thus 
the kernel doesn't think it's bound locally.
  · There are rules that match 192.168.2.* destination IPs and choose 
alternate routing tables as appropriate.
  · The alternate routing tables have device routes pointing to an 
interface.
  · There are rules that match the incoming interface and choose an 
alternate routing table as appropriate.

I would expect any Linux kernel that supports PBR to be able to support 
this.

The only exception might be the /proc tunable to accept local.

> Yes, I'd look into something like this for a BFD-EM implementation.

*nod*

> Those could be used for a relatively simple proof-of-concept.

ACK

> I'd expect that using AF_PACKET, SOCK_RAW removes the need for the hacks 
> used by Project Zero.  They were using e.g. curl instead of writing 
> their own network application.

Agreed.

> Thanks,

:-)



-- 
Grant. . . .
unix || die


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4008 bytes --]