* Full NAT forward and source routing - possible without packet marking? @ 2017-06-30 13:55 Øyvind Kaurstad 2017-07-01 10:05 ` Pascal Hambourg 2017-07-01 20:26 ` Robert White 0 siblings, 2 replies; 23+ messages in thread From: Øyvind Kaurstad @ 2017-06-30 13:55 UTC (permalink / raw) To: netfilter Hi, Here's a quick rundown of the setup in question: I have a device (kernel 4.1.6) with multiple interfaces, and I want to forward outside traffic coming into one interface, and have it exit a different interface. The reply (return traffic) should go in the exact opposite direction. To do this, I am using DNAT and SNAT (masquerading), configured with nftables (v0.5 for now, will update at some point in the future). rp-filtering is turned off for all interfaces. The device has three interfaces (eth0, eth1 and ppp0). The traffic in question first enters ppp0 (from the internet) with the ppp0 address as the destination. The source address can be anything, and not known in advance. Then I have a DNAT-rule to translate the destination address to a device which is behind eth1 (so packet will just be forwarded). I also have masquerading enabled, so the source address is also rewritten when it exits eth1. The reply packet then comes back at eth1, with a destination address that is eth1's address, and a source address that belongs to the replying device. Now the packet will need to have both the source and destination addresses changed, reversing both SNAT/masquerading and DNAT, and then sent back out ppp0. All the packet addresses are correctly rewritten (as expected) so that the final packet has a destination address of the initial internet client and a source address of my ppp0 interface, but the problem is that the packet exits out eth0 (which is the default route) and is then lost. My first attempt at fixing this was to add a new default route out ppp0 in a separate table, and then adding a rule to trigger this whenever the source address was that of the ppp0 interface (which is indeed the source address in the final packet). However, this rule does not trigger, because as far as I can tell, the routing decision is made before the DNAT is reversed, and at that point the source address of the packet is still the address of the replying device. For traffic directed at the router itself, this rule works, and ensures that replies go back out the ppp0 interface. Since I don't know anything about the client's address, I can't add any explicit routes to it, hence I wanted it to be source routed. Which brings me to the question in the subject, is it even possible to have this work without using the features of marking the connection and then mark the return packets and have a routing rule for it? I've tested this, and I can get that to work, but if there is a way to have the source routing work without doing that, I'd like to know. If anyone would care to comment on this I'd be grateful, and if necessary I can of course give some specific examples of my config and sample packet flows. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-06-30 13:55 Full NAT forward and source routing - possible without packet marking? Øyvind Kaurstad @ 2017-07-01 10:05 ` Pascal Hambourg 2017-07-01 20:26 ` Robert White 1 sibling, 0 replies; 23+ messages in thread From: Pascal Hambourg @ 2017-07-01 10:05 UTC (permalink / raw) To: oyvind; +Cc: netfilter Le 30/06/2017 à 15:55, Øyvind Kaurstad a écrit : > > Which brings me to the question in the subject, is it even possible to > have this work without using the features of marking the connection and > then mark the return packets and have a routing rule for it? I don't know about nftables, but with iptables you can avoid marking the connection and use the conntrack match to check the original destination address. However AFAIK you cannot avoid marking the reply packets if, from the routing point of view, nothing distinguishes packets that should be sent to different interfaces. One workaround is to forward incoming packets to alternate addresses. A target device has primary address A and alternate address B. For ordinary traffic, it uses address A. When receiving a packet on ppp0, the routing device forwards it to the alternate address B. The target device sends the reply packet from source address B and the routing device can use source-based routing to route it back to ppp0. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-06-30 13:55 Full NAT forward and source routing - possible without packet marking? Øyvind Kaurstad 2017-07-01 10:05 ` Pascal Hambourg @ 2017-07-01 20:26 ` Robert White 2017-07-01 22:17 ` zrm 2017-07-02 7:29 ` Full NAT forward and source routing " Pascal Hambourg 1 sibling, 2 replies; 23+ messages in thread From: Robert White @ 2017-07-01 20:26 UTC (permalink / raw) To: oyvind, netfilter On 06/30/2017 01:55 PM, Øyvind Kaurstad wrote: > Hi, > ... stuff deleted for brevity ... What you describe is what virtually every router does, that being taking in packets on one port and pushing them back out on another. I think you are over-thinking, and so over-dressing your rules. One does not typically rewrite the packets _leaving_ your router on your local subnets. So, for instance, once you DNAT the incoming packet you _don't_ want to SNAT it. (ASIDE: Note going forward that I use "ext0" or "ext1" etc for my "external", e.g. public facing interfaces-so your ppp0-in my rules. I call my internal bridges "int0" etc, and the physical interfaces "loc0" etc for local.) For example, here's my XBOX and VONAGE rules: iptables --table nat --append PREROUTING --in-interface ext0 --proto udp --match multiport --destination-ports ${VONAGEPORTS} --jump DNAT --to-destination ${VONAGEIP} iptables --table nat --append PREROUTING --in-interface ext0 --proto udp --match multiport --destination-ports ${XBOXLIVEPORTS} --jump DNAT --to-destination ${XBOXLIVEIP} I don't know or care about the real source addresses, ever, because the target devices (my xbox and my vonage router) will just reply to the (default) gateway and the reply goes back where it came from, which is the whole point of connection tracking. So in all cases, any NAT flow (a flow being a TCP connection, or two-way UDP exchange) has a basic structure: I -- R -- E. I == Internal Device. R == Router. E == External Device (the internet). In all cases, the external device (E) thinks it's talking to the router (R), but the internal device (I) always knows it's talking to the external device (E). So R's job is to hide the existence of all devices I from all devices E; but all devices I know and believe they are talking to E directly. Also understand that you are really just dealing with the first packet of any flow. Unless you do some very special work, the second and subsequent packets of a flow wont really even traverse your ruleset as the connection tracker will recognize the response as part of an established flow and automatically and magically perform the reverse magic. So you should _NEVER_ SNAT packets leaving the router onto a private segment, e.g. to devices I. Here's the guide: If the flow starts with a external device E, then it should hit a DNAT rule to translate the _destination_ address to some device I. If the flow starts with an internal device I, then it should hit an SNAT rule (either real SNAT or Masquerade, which is just automatic SNAT so that you don't need to code the actual IP address of the public port). You should always have an established and related rule to catch some of the overhead conditions and let data flow as it should. The second and subsequent packets in a flow are handled by system magic and so don't touch any of your rules in detail. So here's your error: > Then I have a DNAT-rule to translate the destination address to a device which is behind eth1 (so packet will just be forwarded). I also have masquerading enabled, so the source address is also rewritten when it exits eth1. You should _NEVER_ SNAT or Masquerade the internal adapters. It's both harmful and pointless. So your SNAT/Masquerade rule should be limited to the ppp0 device. iptables --table nat --append POSTROUTING --out-interface ext0 --jump SNAT --to-source ${PUBLICIP} ... more to consider ... In the original and "most correct" form, a DMZ, named after the "demilitarized zone" concept, is a semi-public network segment _between_ _two_ _routers_. I -- R1 -- D -- R2 -- E All hosts D appear behind DNAT rules in both R1 and R2. All hosts I can initiate connections to all hosts E _and_ all hosts D, but all hosts E can only initiate connections to all hosts D, and all hosts I are completely invisible to all hosts E. It is now common for people to do a "dogleg DMZ" where the D hosts are on one adapter (say eth1) and the I hosts are on another (say eth0). In this case, the rules work the same but get a little redundant. So let's pick some unambiguous names: extN == a public facing interface. intN == a private interface. dmzN == the dogleg with the common services. $PIP == The well-known IP address that users use to access a service like the web server. $DIP == The actual IP address of the actual web server $DSIP == The IP address of the DMZ adapter. First, that IP address goes on the external interface: ip address add $PIP dev extN And the DMZ adapter gets its IP address ip address add $DSIP dev dmzN Next, all incoming traffic from non-dmz hosts for the DMZ service gets the DNAT treatment: iptables --table nat --append PREROUTING ! --in-interface dmzN --protocol tcp --destination $PIP --destination-port 80 --jump DNAT --to-destination $DIP Note that I exclude/prevent bounce-back on the dmzN port just for completeness. Other than that, if it's my well-know public IP address $PIP and port 80, it goes to the web server at $DIP and that's it. Next, anything leaving via extN needs to appear to have come from the single host $PIP, so we need one SNAT rule. iptables --table nat --append POSTROUTING ! --in-interface extN --out-interface extN --jump SNAT --to-source $PIP See that I prevent bounce-back source-routed attack reflection by refusing to SNAT anything that came in on the same interface it's leaving via. This is a small protection (since you said you turned off rp_filtering, which you really should never need to do). And that's basically it. All hosts, public and private, can find your web server at $PIP:80 and your internal hosts on intN can all reach the internet. (I skipped the ACCEPT rules and all that since this was a discussion of address translation and not policy.) If dmzN and intN interfaces are the same thing, then you just have to connect to the IP address of the service by its true/private IP address instead of your public one (as you don't ever really want to route packets out the same interface they came in on anyway) and that's a problem best addressed by DNS and or /etc/hosts files if you must. A full two-router DMZ is very like the dogleg. The public-side router knows all about the internal network, does SNAT and can only DNAT to the DMZ hosts; and the private side router also DNATs to the DMZ hosts but does no SNAT at all. Then the DMZ hosts need routing information to pick the best host for each flow; this can be done with static tables, dhcp, or redirect messages as you find most convenient. Hope this helps. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-01 20:26 ` Robert White @ 2017-07-01 22:17 ` zrm 2017-07-01 23:50 ` Robert White 2017-07-02 7:29 ` Full NAT forward and source routing " Pascal Hambourg 1 sibling, 1 reply; 23+ messages in thread From: zrm @ 2017-07-01 22:17 UTC (permalink / raw) To: Robert White, oyvind, netfilter On 07/01/2017 04:26 PM, Robert White wrote: > So, for instance, once you DNAT the incoming packet you _don't_ want to > SNAT it. What about hairpin NAT? Suppose you have a port mapping from the router's public IP (2.2.2.2) to some private IP on the LAN (10.2.2.2). Then 2.2.2.2 is published in a rendezvous server and some other device (10.2.2.3) on the same LAN segment learns that address and opens a connection. Now you need the SNAT rule, otherwise the router would translate the packet for 2.2.2.2 to 10.2.2.2 and 10.2.2.2 would send its response to 10.2.2.3. 10.2.2.3 is local so it doesn't pass through the router to be translated back and the connection fails because 10.2.2.3 is expecting a response from 2.2.2.2 rather than 10.2.2.2. It would obviously be better for the applications to use the private addresses directly but you might not be in control of that. So you need to know the in-interface or similar because you should only do the SNAT for hairpin if the client is internal. The interesting question is whether that can be done without marking. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-01 22:17 ` zrm @ 2017-07-01 23:50 ` Robert White 2017-07-03 18:07 ` Hairpin NAT " zrm 0 siblings, 1 reply; 23+ messages in thread From: Robert White @ 2017-07-01 23:50 UTC (permalink / raw) To: zrm, oyvind, netfilter On 07/01/2017 10:17 PM, zrm wrote: > On 07/01/2017 04:26 PM, Robert White wrote: >> So, for instance, once you DNAT the incoming packet you _don't_ want to >> SNAT it. > > What about hairpin NAT? As stated, you really want to do local services by adjusting the DNS unless it's impossible. Sure, you can do hairpin NAT, but it's usually just a bad design since it doubles the network traffic on the one interface. So I understand that there are some odd corner cases that you just cannot fix, such as lying to a black-box appliance, but it's a technique that you _can_ do, but that you have virtually no _good_ reason to ever do. IF you must you want to restrict the hell out of the SNAT so that it only applies to packets that are making that hairpin turn, rather than masquerading/snat all the traffic. So really, if you are in control of the router and the DNS, or you intercept the DNS requests with a REDIRECT rule at the router, you can inject the correct local address anyway and you don't need to hairpin. If you want to put the service at a common public IP address and leave it that way, putting your server on a dogleg/spur is already a much better idea anyway. Why? Because if you have a public service (say a web browser) you don't _want_ it on your local segment. One of the goals (that people often forget) of a firewall is to protect yourself, so if the server is in the free local segment then a compromise of the server is effectively a compromise of the router. If the server(s) are on a dogleg then compromising the server can only give the attacker access to the other servers on the dogleg (if you did your rules right). A server that the public can touch by any means should be isolated from the rest of your private network. For instance, a public-facing web server should never be allowed to make outgoing connections to the rest of your infrastructure. The dogleg rules should, for example, block every form of egress from that server to your real properties or the internet in general. By example, the typical web server exploit causes the web server to make an ssh or telnet connection, or an outbound web request so that the escalated privilege can be used for some purpose. It might also involve putting a web snoop of some sort into the server to watch the incidental traffic for passwords and such. So an actually secure setup would, for example, prevent the web server box from initiating questionable traffic, and would isolate your local SMB type traffic from the wire entirely. So if you have to haripin for some reason, then you should dogleg instead. Period. It's a half-arsed solution to an inherently bad design. A commodity home router doesn't usually offer the necessary hardware to dogleg, but commodity home routers are not designed for Real Services Safety™. Anything you can put a network card in can be built to never need to hairpin anything. So hairpin routing is a bad idea that is popular for bad reasons. The rendezvous server example is fixed by a dogleg, but it's also fixed by having both addresses listed for the server and just NAKing (via REJECT) the public address if its received on the private interface, causing the secondary address to be used. Of course, because of your smart routing the local address should have been chosen for its lower metric to start with, but that's another topic. HOWEVER If you do hairpin, you still don't need to mark any packets or anything. You just use the SNAT/MASQUERADE rule with highly restricted matches just for the hairpin server. An example culled from the internet: -A PREROUTING -d 89.179.245.232/32 -p tcp -m multiport --dports 22,25,80,443 -j DNAT --to-destination 192.168.2.10 -A POSTROUTING -s 192.168.2.0/24 -o ppp0 -j MASQUERADE -A POSTROUTING -s 192.168.2.0/24 -d 192.168.2.10/32 -p tcp -m multiport --dports 22,25,80,443 -j MASQUERADE Notice how the last rule range matches the source, but exactly matches the hairpin target both by exact address and specific ports. Normal non-hairpin traffic simply doesn't get SNAT at all. And still, and again, the connection tracking does all the necessary heavy lifting work with no particularly complex nonsense. It's ugly and insecure, IMHO of course, to leave that server on your general use network like that for the reasons already stated. Spending forty bucks on an extra network card would be _much_ safer and more efficient. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-01 23:50 ` Robert White @ 2017-07-03 18:07 ` zrm 2017-07-04 1:14 ` Robert White 0 siblings, 1 reply; 23+ messages in thread From: zrm @ 2017-07-03 18:07 UTC (permalink / raw) To: Robert White, netfilter The hairpin issue is unrelated to the original one so I'll change the subject line to separate it. On 07/01/2017 07:50 PM, Robert White wrote: > On 07/01/2017 10:17 PM, zrm wrote: >> On 07/01/2017 04:26 PM, Robert White wrote: >>> So, for instance, once you DNAT the incoming packet you _don't_ want to >>> SNAT it. >> >> What about hairpin NAT? > > As stated, you really want to do local services by adjusting the DNS > unless it's impossible. > > Sure, you can do hairpin NAT, but it's usually just a bad design since > it doubles the network traffic on the one interface. > In my case I have a Port Control Protocol daemon so I'm stuck with it. The client software will often fail for internal peers without hairpin and is written by numerous independent third parties. > If you want to put the service at a common public IP address and leave > it that way, putting your server on a dogleg/spur is already a much > better idea anyway. > > Why? > > (...reasons...) > Unfortunately that doesn't get me out of it. My daemon has no control over the physical topology of the user's network, and the internal hosts increasingly _are_ the public-facing hosts. People want to map ports for games, VoIP, IPFS, etc. If you had to put anything that wanted to map any port into the DMZ then most everything would be in the DMZ. :) I feel like isolating public services has always been backwards anyway. An internal network with hard shell with soft middle means that one compromised internal device becomes a big fire. And the modern trend has been for devices to offer public services even without mapped ports, by making outgoing connections to a server that forwards back incoming messages. Everything from Facebook and YouTube to VPNs and Tor is doing this now. It's no different fundamentally from mapping a port except that now everything appears to the router as outgoing TLS or ssh rather than separate ports for different services that could previously have been used to decide whether to pass traffic or not. So you have to assume everything has public services. The only answer now is for the endpoint devices to secure themselves even against other internal hosts, because only they know what the traffic really is. (One could argue it should have been that way the whole time.) The place for externally-imposed network isolation is the naive legacy systems that don't do that properly so they have to be protected from potentially compromised internal hosts that have no business with them. > If you do hairpin, you still don't need to mark any packets or anything. > You just use the SNAT/MASQUERADE rule with highly restricted matches > just for the hairpin server. > > An example culled from the internet: > > -A PREROUTING -d 89.179.245.232/32 -p tcp -m multiport --dports > 22,25,80,443 -j DNAT --to-destination 192.168.2.10 > -A POSTROUTING -s 192.168.2.0/24 -o ppp0 -j MASQUERADE > -A POSTROUTING -s 192.168.2.0/24 -d 192.168.2.10/32 -p tcp -m multiport > --dports 22,25,80,443 -j MASQUERADE That requires knowing which address ranges are internal or not, which I don't. What I have currently is something like this: -A PREROUTING -p udp -d 2.2.2.2 --dport 3456 ! --in-interface external0 -j MARK --or-mark ${HAIRPIN_MARK} -A PREROUTING -p udp -d 2.2.2.2 --dport 3456 -j DNAT --to-destination 10.2.2.2:3456 -A POSTROUTING -m mark --mark ${HAIRPIN_MARK}/${HAIRPIN_MARK} -j SNAT --to-source 2.2.2.2 This seems to do the job, but I would prefer not to use marking in a daemon like this because it's too easy for the administrator to not expect it and then use the same mark for something else. What would be perfect is to have a rule in POSTROUTING with "! -in-interface external0" but --in-interface is not allowed in POSTROUTING. So the question remains is whether it's possible to get the same effect without marking. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-03 18:07 ` Hairpin NAT " zrm @ 2017-07-04 1:14 ` Robert White 2017-07-04 5:48 ` K ` (2 more replies) 0 siblings, 3 replies; 23+ messages in thread From: Robert White @ 2017-07-04 1:14 UTC (permalink / raw) To: zrm, netfilter On 07/03/2017 06:07 PM, zrm wrote: > So the question remains is whether it's possible to get the same effect > without marking. > I have no Good Information™, but I suspect you could do a _stateless_ NAT using ip rule and ip route commands that didn't need any packet marking. I'm not sure if that's a "netfilter" topic, per se, since stateless nat basically avoids the connection tracer and all that, but it looks a little brittle from what I've read. (I've never actually tried it as it doesn't seem to be the best choice.) I've honestly go no clue why you cant use --in-interface in a POSTROUTING chain. I mean obviously --out-interface isn't going to be available in PREROUTING, but I don't see how the packet could have "lost" it's in-interface information at that point. The restriction seems arbitrary (absent some obscure technical detail I don't know that explains it) so I asked the question outright in a separate thread. nft allows the syntax for iif but I'm not sure if it works or not. > I feel like isolating public services has always been backwards > anyway. An internal network with hard shell with soft middle means > that one compromised internal device becomes a big fire. That bit reads as self-contracictory to me. I don't understand what you are saying. The second sentence is exactly why one does the first. So you should have a wall between the outside world and your public services, and then more wall between your public services and your private services and private users. The goal of isolating public services exists because the public can easily find and so more easily compromise a public service. So yea, real security happens in layers. If I crack your web server, you don't want that to put me in the same data flow as, say, your legal documents and company property and future salles leads and customer credit card info and whatever else you might not want to share with the world. It's just basic security, like in the real world. The bank lets lots of people into the lobby, and if a robber compromises the teller he can get the teller to let him into the money-vault that backs the teller. But the robber isn't going to automatically get access to _all_ the banks vaults everywhere, or the HR records, or the list of contracts the bank holds, or all the mortgage information. Those things simply are not kept where the teller can reach them during a robbery. So in a good network design you keep the things that the public can touch in any way on a segment that isn't the same as the minute-by-minute details of your whole business. So you set up your firewall around your web service and you screw it down tight. If the public can only use http and https then clearly that's all you allow from the public net. And if the web server will only send back answers on those sessions, then you don't let the web server initiate _any_ connections from itself to the internet. If the web server has to talk to a trivial database, maybe you leave that on the same hardware, or maybe you colocate it on the network. But if the web server needs to talk to a critical database, like one full of customer records and credit cards, then you segregate them and only let the necessary ports through. And again, that database server is not allowed to talk to the rest of the world. Then if you need to maintain those servers you have your private segment(s) and they are allowed to initiate ssh sessions into those servers, but those servers are not allowed to ssh back towards the more private network. And it's the little things that make a system like this work correctly. The ideal front-most router is smooth on both sides, with no SSH or admin ports available at all. That front server is maybe only available by direct console attachment. Or maybe you spend eighty bucks on an extra pair of network cards and you allow remote administration, but only from the network segment of the dedicated maintenance link. If you don't have the cash, then you only allow the "Back router" talk directly to the front router. If you don't have enough money to put two separate routers with a proper DMZ between them, then you do the dogleg DMZ off of a single router. But basically a good security design treats every public-accessible server as if it's already been compromised and so as if you cannot trust it's physical connections at all. One of the most paranoid systems I ever designed had a front-side router/firewall that had no writable storage on it at all. It booted from a CD which had the kernel and initramfs. That initramfs had all the firewall rules and modules and whatnot and if it needed to be altered you'd just burn a new CD and swap them out. What little logging there was went out a separate ethernet device as syslog events (udp broadcasts). Behind that router was another router/firewall that had four adapters. One for corporate innards. one for the public facing services. One for the database server(s) that supported the public services. And one for monitoring the forward firewall's logs. No system is absolutely uncrackable, of course, but if an external hacker wanted to compromise the corporate information he'd have to end up crafting a web request that caused the web server to craft a database request that caused the database server to lie-in-wait for an admin to connect to it and take over that connection in a way that would cause the admin's computer to send something back to the attacker. This is "not bloodly likely". So compromising the web server wouldn't for example, let the web server to make an scp or ftp (or whatever) session to export it's data, the compromise would have to export the data back through that self-same tcp session. Dont mistake "all services" for "public services". Behind that second firewall the company was just as stupid as most companies. They had people sharing segments of their hard drives. Pooled servers with just ludicrously broad write policies, printers, store and forward scanners, all the normal stupid things that let business function. And you know, what, its well they should. Security that becomes a denial of service attack on the corporation's innards just encourages misuse. If someone was going to steal that stuff there were going to have to come on sight and do it from there. But that web portal was not going to be the way in. And that's as much as you can hope for really. So yes, sure, individual devices need to be as secure as reasonably possible. And yes, you can't stop an internal (on site) actor. But you sure-as-shooting can make it virtually impossible for an outside actor to "climb up" your public services by simply not letting those public service providers do _any_ of that outgoing nonsense. Your web server has no business initiating an SSH session to anything. That's not it's job. If a hacker gets control of that machine he's got no problem turning off it's internal rules. But if walks its entire internal security to root privilige, only to find out that the web server is "alone" on its segment and none of the routers will let it do _anything_ but listen to time broadcasts, send database queries through a pinhole to an equally sequestered database farm, and reply to the TCP packets it received on the HTTP and HTTPS ports... he will be sad. Network isolation not about (just) "naive legacy servers", it's about the core principles of security. Sure individual offices have locks on their doors, and individual desks have locks on their drawers, but if you say the big locks on the front doors and the individual security gates throughout the campus are just for "weak legacy desks" you've utterly missed the point. The key phrase you should ponder is "Defence in Depth". The little locks just slow people down once they are inside the building. It's the big locks on the well designed outer doors that do the most good. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-04 1:14 ` Robert White @ 2017-07-04 5:48 ` K 2017-07-04 7:07 ` Neal P. Murphy 2017-07-04 20:53 ` Pascal Hambourg 2017-07-08 19:54 ` zrm 2 siblings, 1 reply; 23+ messages in thread From: K @ 2017-07-04 5:48 UTC (permalink / raw) To: netfilter What do all the locks in the world help when you invite the burglar in for tea? In other words: most IT departments have the incoming traffic pinned down as you described, but a single executable disguised as a clip of a cute kitty, downloaded and executed by any employee is what nowadays forms the real threat. On July 4, 2017 3:14:59 AM GMT+02:00, Robert White <rwhite@pobox.com> wrote: >They had >people sharing segments of their hard drives. Pooled servers with just >ludicrously broad write policies, printers, store and forward scanners, >all the normal stupid things that let business function. And you know, >what, its well they should. Security that becomes a denial of service >attack on the corporation's innards just encourages misuse. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-04 5:48 ` K @ 2017-07-04 7:07 ` Neal P. Murphy 2017-07-04 10:21 ` Robert White 0 siblings, 1 reply; 23+ messages in thread From: Neal P. Murphy @ 2017-07-04 7:07 UTC (permalink / raw) Cc: netfilter On Tue, 04 Jul 2017 07:48:36 +0200 K <netfilter@quintux.com> wrote: > What do all the locks in the world help when you invite the burglar in for tea? In other words: most IT departments have the incoming traffic pinned down as you described, but a single executable disguised as a clip of a cute kitty, downloaded and executed by any employee is what nowadays forms the real threat. And that's why I maintain that SSL/TLS is the one of the worst things that could've happened to The Internet: our peripheral firewalls are powerless to prevent malware from traversing conns encrypted with SSL/TLS. Neal > > On July 4, 2017 3:14:59 AM GMT+02:00, Robert White <rwhite@pobox.com> wrote: > >They had > >people sharing segments of their hard drives. Pooled servers with just > >ludicrously broad write policies, printers, store and forward scanners, > >all the normal stupid things that let business function. And you know, > >what, its well they should. Security that becomes a denial of service > >attack on the corporation's innards just encourages misuse. > -- > To unsubscribe from this list: send the line "unsubscribe netfilter" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-04 7:07 ` Neal P. Murphy @ 2017-07-04 10:21 ` Robert White 0 siblings, 0 replies; 23+ messages in thread From: Robert White @ 2017-07-04 10:21 UTC (permalink / raw) To: Neal P. Murphy; +Cc: netfilter On 07/04/2017 07:07 AM, Neal P. Murphy wrote: > On Tue, 04 Jul 2017 07:48:36 +0200 > K <netfilter@quintux.com> wrote: > >> What do all the locks in the world help when you invite the burglar in for tea? In other words: most IT departments have the incoming traffic pinned down as you described, but a single executable disguised as a clip of a cute kitty, downloaded and executed by any employee is what nowadays forms the real threat. > > And that's why I maintain that SSL/TLS is the one of the worst things that could've happened to The Internet: our peripheral firewalls are powerless to prevent malware from traversing conns encrypted with SSL/TLS. Well communism would be the perfect form of government if no humans were involved... but humans are involved. Humans are the week point in any system. Imagining you can fix human stupidity by just tightening the screws a little tighter is just fairy dust and unicorn farts. The easiest way to breach a network is to drop thumb drives, cdroms, and urls in the parking lot. Banning encryption isn't going to save a single thing. Make passwords to onerous and they get written on post-it notes. Ban usb drives and people plug in media players. Ban all USB and people will fax shit. Technology cannot fix human engineering. So you build the system to be hard enough for remote nonsense to be easy and then lose the superman complex. Think you can stop things using a ban on TLS? I scoff at you. I'll rot-13 the virus and send it with instructions. I'll shar that puppy and send it with instructions. Hell, I'll just send instructions in plain text and _someone_ in your office will be stupid enough to do _all_ the work. Heck for all your talk I'd bet money that you've never done a full system restore drill on you computers at home or at the office, and at least half of your data isn't even backed up at all,let alone off-site. A system too onerous to use _will_ be subverted. Every time. Period. The only way to keep a computer safe is to turn it off, unplug it, and shred it. So you need to concentrate on doing the best things to stop the most damage. For all that a virus scanner can stop a virus, maybe, if it's already been told about the virus, the most pernicious encoding of a bad idea is a well-phrased meme handed to a neophyte. You can't fix stupid and you cant's solve security through draconian technological intervention. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-04 1:14 ` Robert White 2017-07-04 5:48 ` K @ 2017-07-04 20:53 ` Pascal Hambourg 2017-07-04 21:20 ` Neal P. Murphy 2017-07-05 5:28 ` Robert White 2017-07-08 19:54 ` zrm 2 siblings, 2 replies; 23+ messages in thread From: Pascal Hambourg @ 2017-07-04 20:53 UTC (permalink / raw) To: Robert White, netfilter Le 04/07/2017 à 03:14, Robert White a écrit : > > I've honestly go no clue why you cant use --in-interface in a > POSTROUTING chain. Because the POSTROUTING chains also see packets that are generated locally and have no input interface. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-04 20:53 ` Pascal Hambourg @ 2017-07-04 21:20 ` Neal P. Murphy 2017-07-05 5:28 ` Robert White 1 sibling, 0 replies; 23+ messages in thread From: Neal P. Murphy @ 2017-07-04 21:20 UTC (permalink / raw) Cc: netfilter On Tue, 4 Jul 2017 22:53:11 +0200 Pascal Hambourg <pascal@plouf.fr.eu.org> wrote: > Le 04/07/2017 à 03:14, Robert White a écrit : > > > > I've honestly go no clue why you cant use --in-interface in a > > POSTROUTING chain. > > Because the POSTROUTING chains also see packets that are generated > locally and have no input interface. Logically, (for example) locally generated packets (that have no input interface) and packets from eth1 should equally fail to match "--in-interface eth2". In other words, a packet that has no source interface should never match any '--in-interface X' option because it clearly did not, and could not have, come in from interface X. Of course, the implementation may present different conditions and limitations. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-04 20:53 ` Pascal Hambourg 2017-07-04 21:20 ` Neal P. Murphy @ 2017-07-05 5:28 ` Robert White 1 sibling, 0 replies; 23+ messages in thread From: Robert White @ 2017-07-05 5:28 UTC (permalink / raw) To: Pascal Hambourg, netfilter On 07/04/2017 08:53 PM, Pascal Hambourg wrote: > Le 04/07/2017 à 03:14, Robert White a écrit : >> >> I've honestly go no clue why you cant use --in-interface in a >> POSTROUTING chain. > > Because the POSTROUTING chains also see packets that are generated > locally and have no input interface. Wouldn't that just be a mismatch then? Equivalent to a NULL (in the database usage of the term) where all tests against a missing interface simply fail. In the alternate, locally generated packets would seem to come from the loop/local interface. I suppose the mismatch logic could be expensive though, I don't know where this code base's relative efficiencies lie. Of course this idea may be well-travelled and settled ground. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-04 1:14 ` Robert White 2017-07-04 5:48 ` K 2017-07-04 20:53 ` Pascal Hambourg @ 2017-07-08 19:54 ` zrm 2017-07-08 21:55 ` Robert White 2 siblings, 1 reply; 23+ messages in thread From: zrm @ 2017-07-08 19:54 UTC (permalink / raw) To: Robert White, netfilter On 07/03/2017 09:14 PM, Robert White wrote: > I have no Good Information™, but I suspect you could do a _stateless_ > NAT using ip rule and ip route commands that didn't need any packet > marking. I'm not sure if that's a "netfilter" topic, per se, since > stateless nat basically avoids the connection tracer and all that, but > it looks a little brittle from what I've read. (I've never actually > tried it as it doesn't seem to be the best choice.) I looked into that at one point for other reasons, but it won't translate ports, and according to the man page ip-route nat seems to have been removed anyway. > I've honestly go no clue why you cant use --in-interface in a > POSTROUTING chain. I mean obviously --out-interface isn't going to be > available in PREROUTING, but I don't see how the packet could have > "lost" it's in-interface information at that point. The restriction > seems arbitrary (absent some obscure technical detail I don't know that > explains it) so I asked the question outright in a separate thread. > > nft allows the syntax for iif but I'm not sure if it works or not. I'll have to test nft at some point. I expect to switch entirely to libnftnl once distributions that don't have it have well died out but until then it's simpler to use iptables everywhere. >> I feel like isolating public services has always been backwards >> anyway. An internal network with hard shell with soft middle means >> that one compromised internal device becomes a big fire. > > That bit reads as self-contracictory to me. I don't understand what you > are saying. The second sentence is exactly why one does the first. What I'm saying is that it's better to isolate sensitive or vulnerable systems from everything possible than trying to isolate everything possible from potentially compromised systems. Any system could be compromised. A public web server is not significantly more likely to be than a user's laptop. > So you should have a wall between the outside world and your public > services, and then more wall between your public services and your > private services and private users. > > The goal of isolating public services exists because the public can > easily find and so more easily compromise a public service. > > So yea, real security happens in layers. > > If I crack your web server, you don't want that to put me in the same > data flow as, say, your legal documents and company property and future > salles leads and customer credit card info and whatever else you might > not want to share with the world. My argument is that public is not particularly rare or special. It can make sense to e.g. isolate finance from advertising. And if there isn't any _reason_ for the web server to talk to the finance systems then by all means isolate them from each other too. You're probably right that a public server is more likely to be compromised than an equally well maintained internal server, but it doesn't fare so well compared to _user devices_. A random laptop is already exposed to browser vulnerabilities, email attachments, third party wifi, etc. If it becomes "public" due to a port mapped for VoIP or similar, that is not going to change the security posture significantly. A non-zero number of the devices are likely to be infected regardless and the same precautions should be taken either way. So if the traditional model was to separate public servers from internal servers and user devices and then lock down the public servers, I would argue a better model is to separate public servers and user devices from internal servers and then lock down the internal servers. > So you set up your firewall around your web service and you screw it > down tight. If the public can only use http and https then clearly > that's all you allow from the public net. And if the web server will > only send back answers on those sessions, then you don't let the web > server initiate _any_ connections from itself to the internet. Once the attacker has control over the server, none of that really matters. Making the attacker have to export the data back through that self-same tcp session is just an inconvenience when they can do just that. Network-imposed default deny, especially for outgoing, also tends to produce false positives, and that has security implications too. For example, if you prohibit all outgoing connections, can the server get updates? Send alert messages? But opening only the HTTP[S] ports for updates (or any other reason) falls into the ecological damage trap. If a critical mass of networks block everything but HTTP then everything will just use HTTP, which only makes everything slower, more complicated and less secure than it should be. And defeats its own purpose because default deny with HTTP open becomes equivalent to opening everything -- including things you might have wanted to explicitly reject. Better to reject things you know you want to reject and just log things you don't expect. Because rejecting things you don't expect only forces everything to look like the things you do expect, which in the end makes it impossible to distinguish anything when you do want to explicitly reject something. > Sure individual offices have locks on their doors, and individual desks > have locks on their drawers, but if you say the big locks on the front > doors and the individual security gates throughout the campus are just > for "weak legacy desks" you've utterly missed the point. > > The key phrase you should ponder is "Defence in Depth". > > The little locks just slow people down once they are inside the > building. It's the big locks on the well designed outer doors that do > the most good. In practice poor exterior security is common; anyone can show up at 9AM and walk through the door behind 50 other people. Businesses with walk-in customers generally don't even lock the exterior doors. Moreover, sensitive information like HR and finance records is kept double-locked and access-restricted even inside the building. And the main deterrent to physical intrusion isn't locks, it's the law. That doesn't work for network security because the attacker is outside your jurisdiction and behind seven proxies. Security as a self-DoS is a thing but that's the point. It's more practical to store your sensitive business records in a safe and keep unauthorized people out of the safe than to store them in the driveway and have to keep everyone out of the campus. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Hairpin NAT - possible without packet marking? 2017-07-08 19:54 ` zrm @ 2017-07-08 21:55 ` Robert White 0 siblings, 0 replies; 23+ messages in thread From: Robert White @ 2017-07-08 21:55 UTC (permalink / raw) To: zrm, netfilter On 07/08/2017 07:54 PM, zrm wrote: > On 07/03/2017 09:14 PM, Robert White wrote: > I looked into that at one point for other reasons, but it won't > translate ports, and according to the man page ip-route nat seems to > have been removed anyway. man ip-rule Stateless nat is a rule not a route. You end up needing both the rule and some route stuff to make it work. And being "stateless" it, indeed, cannot track packet mappings as knowing what to do to ports on the return path requires state. > My argument is that public is not particularly rare or special. > > It can make sense to e.g. isolate finance from advertising. And if there > isn't any _reason_ for the web server to talk to the finance systems > then by all means isolate them from each other too. > > You're probably right that a public server is more likely to be > compromised than an equally well maintained internal server, but it > doesn't fare so well compared to _user devices_. > > A random laptop is already exposed to browser vulnerabilities, email > attachments, third party wifi, etc. If it becomes "public" due to a port > mapped for VoIP or similar, that is not going to change the security > posture significantly. A non-zero number of the devices are likely to be > infected regardless and the same precautions should be taken either way. > > So if the traditional model was to separate public servers from internal > servers and user devices and then lock down the public servers, I would > argue a better model is to separate public servers and user devices from > internal servers and then lock down the internal servers. You mistake a design to prevent network intrusion as if it were offered as panacea. It is always wrong to discard good design because it only covers what it's designed to cover. So yes, you need to have other protections from other sources of compromise, but other sources of compromise aren't at issue in the material being addressed. I already mentioned leaving CDROMS and thumbdrives lying around, and other issues as very good attacks. Also in the news: techniques to minimize heart disease may not be applicable to liver flukes... So describing "The traditional model" as you have misses and oversimplifies and conflates too much to be anything other than a straw man because your "if the traditional model is" predicate doesn't actually describe the traditional model correctly. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-01 20:26 ` Robert White 2017-07-01 22:17 ` zrm @ 2017-07-02 7:29 ` Pascal Hambourg 2017-07-02 10:33 ` Robert White 1 sibling, 1 reply; 23+ messages in thread From: Pascal Hambourg @ 2017-07-02 7:29 UTC (permalink / raw) To: Robert White; +Cc: netfilter Le 01/07/2017 à 22:26, Robert White a écrit : > On 06/30/2017 01:55 PM, Øyvind Kaurstad wrote: > > ... stuff deleted for brevity ... Ditto. I think you missed the point and focused on an irrelevant detail in the OP. This topic is about advanced routing. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-02 7:29 ` Full NAT forward and source routing " Pascal Hambourg @ 2017-07-02 10:33 ` Robert White 2017-07-02 11:19 ` Pascal Hambourg 0 siblings, 1 reply; 23+ messages in thread From: Robert White @ 2017-07-02 10:33 UTC (permalink / raw) To: Pascal Hambourg; +Cc: netfilter On 07/02/2017 07:29 AM, Pascal Hambourg wrote: > Le 01/07/2017 à 22:26, Robert White a écrit : > I think you missed the point and focused on an irrelevant detail in the > OP. This topic is about advanced routing. Read the OP again. Traffic is coming in on ppp0, hitting a server hanging off eth1, and then the replies are supposed to go back to eth1 and go out on ppp0. There is literally no simpler task in DNAT. This was not an "advanced routing" question, this was an "introduction to DNAT" level question. It is a mistake to assume that simple questions are complex. OP needed to be shown that he was "overthinking" his rules. ZRM and I discussed hairpin nat in a later fragment of the conversation that almost certainly didn't help Mr. Kaurstad in the slightest. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-02 10:33 ` Robert White @ 2017-07-02 11:19 ` Pascal Hambourg 2017-07-02 15:58 ` Øyvind Kaurstad 0 siblings, 1 reply; 23+ messages in thread From: Pascal Hambourg @ 2017-07-02 11:19 UTC (permalink / raw) To: Robert White; +Cc: netfilter Le 02/07/2017 à 12:33, Robert White a écrit : > On 07/02/2017 07:29 AM, Pascal Hambourg wrote: >> Le 01/07/2017 à 22:26, Robert White a écrit : >> I think you missed the point and focused on an irrelevant detail in the >> OP. This topic is about advanced routing. > > Read the OP again. Traffic is coming in on ppp0, hitting a server > hanging off eth1, and then the replies are supposed to go back to eth1 > and go out on ppp0. > > There is literally no simpler task in DNAT. > > This was not an "advanced routing" question, this was an "introduction > to DNAT" level question. Again, you missed the important point : reply packets must go out through ppp0 but the default route is through eth0, not ppp0. Without advanced routing, reply packets will go out through eth0. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-02 11:19 ` Pascal Hambourg @ 2017-07-02 15:58 ` Øyvind Kaurstad 2017-07-02 17:10 ` Pascal Hambourg 2017-07-02 21:10 ` Robert White 0 siblings, 2 replies; 23+ messages in thread From: Øyvind Kaurstad @ 2017-07-02 15:58 UTC (permalink / raw) To: netfilter Pascal Hambourg wrote: > Le 02/07/2017 à 12:33, Robert White a écrit : >> On 07/02/2017 07:29 AM, Pascal Hambourg wrote: >>> Le 01/07/2017 à 22:26, Robert White a écrit : >>> I think you missed the point and focused on an irrelevant detail in the >>> OP. This topic is about advanced routing. >> >> Read the OP again. Traffic is coming in on ppp0, hitting a server >> hanging off eth1, and then the replies are supposed to go back to eth1 >> and go out on ppp0. >> >> There is literally no simpler task in DNAT. >> >> This was not an "advanced routing" question, this was an "introduction >> to DNAT" level question. > > Again, you missed the important point : reply packets must go out > through ppp0 but the default route is through eth0, not ppp0. Without > advanced routing, reply packets will go out through eth0. First, let me say I'm very grateful to all of you that has taken the time to answer me, and I will study the rather comprehensive replies in detail. I'm clearly no expert, but I'm trying to learn so the information is appreciated. To perhaps clarify a few things: My device is not meant to be a system (or full blown) router per se. It is a specialized custom device that for some cases needs to be able to act as a router. It does not act as a DNS server, and in most cases there will be other routers in the system (at least on the eth0 network, and my device is usually set up as a DHCP client on eth0). The device behind eth1 is not guaranteed to have my device as it's default route (in fact it will most likely not), which is why I believe I need SNAT/masquerading to have it send the replies back to my device. But for the sake of argument, let's assume that the device will send the replies back to my device at eth1 even with SNAT off. I'll try to use an example with fictious IP-addresses to clarify: Assume ppp0 has an address of 10.10.10.10 Assume eth0 has an address of 192.168.10.10 Assume eth1 has an address of 192.168.100.100 Assume the device behind eth1 has an address of 192.168.100.200 Say that I want to forward inbound (UDP or TCP) traffic destined for 10.10.10.10:1111 to 192.168.100.200:1111, so I add a DNAT rule for that. Then when a packet comes in on ppp0 from 100.100.100.100:2222, and my DNAT-rule rewrites the destination address to 192.168.100.200:1111. This packet then leaves eth1 for the device, and the device replies. Without SNAT, the addresses in the reply are: Dst: 100.100.100.100:2222 Src: 192.168.100.200:1111 The packet is then routed, and after the routing decision the DNAT is reversed, so when the packet actually leaves my device, it has these addresses: Dst: 100.100.100.100:2222 Src: 10.10.10.10:1111 However, since the routing decision is made *before* the source address has been rewritten, this packet leaves out eth0, where it is lost. If I have SNAT/masquerading, the reply from the device has these addresses: Dst: 192.168.100.100:2222 Src: 192.168.100.200:1111 The SNAT is then reversed for the packet: Dst: 100.100.100.100:2222 Src: 192.168.100.200:1111 And now the routing happens, but again the packet takes the default route out eth0 and is lost. In none of these cases my source routing rule is hit, because the source address of my ppp0 device is not being rewritten until *after* the routing decision has been made. I could of course add a explicit source routing rule for any packets coming from the external device to go out ppp0, but that's undesirable, because I also want to allow incoming traffic on eth0 to talk to this device (again using DNAT/port forwarding). In this case the replies will work without any special routing rules, since eth0 is in fact the default route. Not sure if this clarified anything, but it still seems to me I need to leverage the connection tracking with packet marking to be able to ensure the reply packets that should go back out a non-default route actually does that. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-02 15:58 ` Øyvind Kaurstad @ 2017-07-02 17:10 ` Pascal Hambourg 2017-07-02 23:20 ` Øyvind Kaurstad 2017-07-02 21:10 ` Robert White 1 sibling, 1 reply; 23+ messages in thread From: Pascal Hambourg @ 2017-07-02 17:10 UTC (permalink / raw) To: oyvind, netfilter Le 02/07/2017 à 17:58, Øyvind Kaurstad a écrit : > > Not sure if this clarified anything There was no need to clarify anything to me. Your original post was clear enough, except the reason for the internal SNAT that you explained but which is irrelevant, as you mentionned. However, hopefully that will help other readers concentrate on the real issue. > but it still seems to me I need to leverage the connection tracking > with packet marking to be able to ensure the reply packets that should > go back out a non-default route actually does that. I'm afraid so, unless you can add a second IP address to the target device. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-02 17:10 ` Pascal Hambourg @ 2017-07-02 23:20 ` Øyvind Kaurstad 0 siblings, 0 replies; 23+ messages in thread From: Øyvind Kaurstad @ 2017-07-02 23:20 UTC (permalink / raw) To: netfilter >> but it still seems to me I need to leverage the connection tracking >> with packet marking to be able to ensure the reply packets that should >> go back out a non-default route actually does that. > > I'm afraid so, unless you can add a second IP address to the target device. The target device is usually a device that I have no control over, and in many cases it can be a device that doesn't even have the capability to have multiple IP addresses on the physical interface. So I'll have to treat it as a one-address blackbox. Still feeling quite happy now, because quite a few lessons has been learned about the inner workings of routing (and netfilter/nftables/iptables) in Linux, so I will be better equipped to tackle future requirements for my device (and other Linux-based devices). For others that read this later on, while there are lots of docs on nftables/netfilter, I came across this one as particularly useful: https://pelican.craoc.fr/ I can't attest that it is correct in all aspects, but I liked the diagram he drew based on the tests he did. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-02 15:58 ` Øyvind Kaurstad 2017-07-02 17:10 ` Pascal Hambourg @ 2017-07-02 21:10 ` Robert White 2017-07-02 23:09 ` Øyvind Kaurstad 1 sibling, 1 reply; 23+ messages in thread From: Robert White @ 2017-07-02 21:10 UTC (permalink / raw) To: oyvind, netfilter On 07/02/2017 03:58 PM, Øyvind Kaurstad wrote: > My device is not meant to be a system (or full blown) router per se. It > is a specialized custom device that for some cases needs to be able to > act as a router. It does not act as a DNS server, and in most cases > there will be other routers in the system (at least on the eth0 network, > and my device is usually set up as a DHCP client on eth0). That's as "router-ish" as routers get. Don't think of it as "not a full-blown router". It may be expected to only have a light packet load but that doesn't make it anything less than a router. First I am going to juggle your addresses a little to get your device (and all of eth1), that is the inner protected network ("PRO") out of the same 192.168.0.0/16 chunk as your "General use network" (e.g. "GUN", e.g. whatever is connected to eth0). This is just to prevent mental typos as I continue. So address ranges: GUN: 192.168.0.0/16 PRO: 172.16.0.0/12 INTERNET: *whole world otherwise So within your router everything going towards 192.168.0.0/16 will go out eth0, and everything going to 172.16.0.0/12 is going to go out eth1. And your problem is that INTERNET is both on ppp0 and some dominant router somewhere in 192.168.0.0/16 (out eth0 somewhere), yes? First notice that this problem, where the internet is in two places, exists regardless of the NAT. So the problem isn't NAT related, its pure routing table stuff. Basically, if the flow came _in_ ppp0 you want it to go _out_ ppp0. This is where people were talking about marking the packets/flow but then what? (DISCLAIMER: I don't have a test rig for this so I'm just typing stuff. There may be typos or errors, but this _should_ be just about right. Now to carry on...) So first, we want to use connection marking. That is, we want any mark we make on any packet something that will live as long as we consider that flow valid. iptables --table mangle --insert 1 PREROUTING --jump CONNMARK --restore-mark iptables --table mangle --append POSTROUTING --jump CONNMARK --save-mark So now any marks we ever put on any packet will be automatically applied to the response packets. Now we want to mark any connection that starts life in from ppp0. The only hard-and-fast requirement is that this marking rule below must happen _after_ the "restore-mark" rule above. So it could go in FORWARD or POSTROUTING or whatever. iptables --table mangle --append PREROUTING --in-interface ppp0 --jump MARK --set-mark 1 --ALTERNATELY-- iptables --table mangle --append PREROUTING --in-interface ppp0 --match conntrack --ctstate NEW --jump MARK --set-mark 1 The latter is more precise by only explicitly marking new connections, but it is also probably not necessary to be that precise because that in-interface condition is very strong. You are also going to want to have a DNAT rule for arriving packets and an SNAT/MASQUERADE rule for ppp0, but again, don't bother with any SNAT from for 192.168.0.0/16 (etho0) or 172.16.0.0/12 (eth1). ASIDE: if your router thing is supposed to front-end/hide it's interior geometry from all comers, you might want a DNAT rule on eth0 and ppp0. Then the DHCP address received by the router on 192.168.X.X will appear to be your protected device. # Port 80 transparently forwareded to protected device iptables --table nat --append PREROUTING ! --in-interface eth1 --protocol tcp --dport 80 --jump DNAT 172.16.x.x # anything originating from the device is SNAT on all _other_ ports. iptables --table nat --append POSTROUTING --in-interface eth1 --jump MASQUERADE (other ports, protocols, and policies work the same way as 80 etc) At this point we are done with iptables. We've directed the traffic and we've marked the flow. Now it's pure routing rules. We need to move the default route to the ppp0 adapter for the marked packets. So pick a number, I've picked "500", we are going to use this number to create a routing rule table. Lets give that number a name so that we don't have to deal with the number. This is a persistent setting so don't put it in your startup scripts. Just do it once as root. # echo "500 ppp0_routes" >> /etc/iproute2/rt_tables Now very early after each/any startup you need to add a rule that says "for all packets marked with a "1" I want to use some specific routing information, adn that's going to be in that table we just mentioned. ip rule add fwmark 1 table ppp0_routes (For clarity, the condition is "fwmark 1" so for all unmarked packets, from unmarked connections, or packets marked with any other number, the "normal default" route will apply because this ppp0_routes table will be skipped completely.) Whenever ppp0 starts up, you'll need to add the default route going out ppp0 for all the packets that have been marked. So in your "ifup" or "post_up" script/function for ppp0 you want to add ip route add default via $NEXTHOP dev ppp0 table ppp0_routes "via $NEXTHOP" may not be necessary depending, but if it is NEXTHOP is whatever default router the PPP connection startup says you should use. Check your pppd documentation for how to collect this information, and be sure to use the "no default route" option for pppd itself as you don't want pppd to overwrite the default route you got from dhcp. This route will _disappear_ whenever ppp0 goes down, which makes cleanup automatic, and is also why you need to put this command in the ifup/pos_up script to put it back. And you are done... (give or take whatever errors I made since, like I said, I have no means to test this here.) SECOND DISCLAIMER: I haven't messed around with PPP in probably eight years or so, so I don't know if you'll need to put more rules in the ppp0_routes list. In particular if your eth1 connection is not direct, you might want to put in a route for 172.16.X.X or whatever. I don't think you will need to, but I recall PPP to be sort of a big jerk. At this point: All sessions started from INTERNET coming in through the normal infrastructre (eth0) is going to go back out the GUN/infrastructure default route. All sessions started from the GUN infrastructure itself (192.168.X.X) will go out to the GUN infrastructure "naturally". All sessions started by the protected device or anything on the PRO network will go out via the GUN. All sessions started by a packet coming in through the ppp0 link will use the ppp default router as their default. SO TO RECAP: (1) it's a route selection problem not a NAT issue. (2) rule based routing does the heavy lifting. (3) packet marking lets you easily activate the alternate rule, and so the alternate default router, without touching the outcome for any other packets. (4) you may have to fiddle with some stuff because my suggestion is untested stuff from memory. 8-) > Not sure if this clarified anything, but it still seems to me I need to > leverage the connection tracking with packet marking to be able to > ensure the reply packets that should go back out a non-default route > actually does that. It has actually clarified a lot. Sorry I misunderstood. I thought your eth0 was the transport under your PPP session (As in ppp-over-ethernet. My bad. 8-) As shown, packet marking and connection tracking is the easiest way to go. You might be able to craft some ip rules to avoid the marking but it's not worth it when the marking solution is so simple. Performance impact should be trivial and limited to the ppp0 link anyway. Again, apologies. Being told I missed "something" wasn't as helpful as being told what thing I missed. /D'oh ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: Full NAT forward and source routing - possible without packet marking? 2017-07-02 21:10 ` Robert White @ 2017-07-02 23:09 ` Øyvind Kaurstad 0 siblings, 0 replies; 23+ messages in thread From: Øyvind Kaurstad @ 2017-07-02 23:09 UTC (permalink / raw) To: netfilter >> My device is not meant to be a system (or full blown) router per se. It >> is a specialized custom device that for some cases needs to be able to >> act as a router. It does not act as a DNS server, and in most cases >> there will be other routers in the system (at least on the eth0 network, >> and my device is usually set up as a DHCP client on eth0). > > That's as "router-ish" as routers get. Don't think of it as "not a > full-blown router". It may be expected to only have a light packet load > but that doesn't make it anything less than a router. I get what you're saying, and I don't really disagree, it's just that on this device there's still quite a few things missing (on purpose) that a "full blown" router has. Hence I said what I did. > First I am going to juggle your addresses a little to get your device > (and all of eth1), that is the inner protected network ("PRO") out of <detailed explanation snipped for brevity> Thanks for writing down all of that. It is useful, even though I actually had most of it down working already. As also mentioned in the subject, the real point of this thread was to find out if packet marking is actually the best/easiest way to solve my problem, and both you and others have confirmed that now. I'm also using nftables (as mentioned in the OP), so the rules you used in iptables are different than mine. The principles are the same though (and netfilter is of course still the underlying framework), so your answer is still useful as a guide to where and when. I've struggled with this for quite some time, and it is a relief to finally have a better grip on it. > Again, apologies. Being told I missed "something" wasn't as helpful as > being told what thing I missed. /D'oh No apologies needed. You've been very helpful, to misunderstand is only human. :-) ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2017-07-08 21:55 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-06-30 13:55 Full NAT forward and source routing - possible without packet marking? Øyvind Kaurstad 2017-07-01 10:05 ` Pascal Hambourg 2017-07-01 20:26 ` Robert White 2017-07-01 22:17 ` zrm 2017-07-01 23:50 ` Robert White 2017-07-03 18:07 ` Hairpin NAT " zrm 2017-07-04 1:14 ` Robert White 2017-07-04 5:48 ` K 2017-07-04 7:07 ` Neal P. Murphy 2017-07-04 10:21 ` Robert White 2017-07-04 20:53 ` Pascal Hambourg 2017-07-04 21:20 ` Neal P. Murphy 2017-07-05 5:28 ` Robert White 2017-07-08 19:54 ` zrm 2017-07-08 21:55 ` Robert White 2017-07-02 7:29 ` Full NAT forward and source routing " Pascal Hambourg 2017-07-02 10:33 ` Robert White 2017-07-02 11:19 ` Pascal Hambourg 2017-07-02 15:58 ` Øyvind Kaurstad 2017-07-02 17:10 ` Pascal Hambourg 2017-07-02 23:20 ` Øyvind Kaurstad 2017-07-02 21:10 ` Robert White 2017-07-02 23:09 ` Øyvind Kaurstad
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.