Surprising CONNMARK behaviors

Surprising CONNMARK behaviors

[Glen Huang]

Hi, all,

I have some CONNMARK rules that only partially behave as intended. I think either my understanding of how CONNMARK works is wrong or the packets I’m sending are idiosyncratic. Would really appreciate it if someone could shed some light.

I’m trying to create a transparent proxy using nginx that functions as a tcp load balancer. Whenever a client connects to the proxy, the proxy sends to/receives from the upstream server using client IP as the source/destination IP. So so that the client IP shows up in upstream logs instead of the proxy’s. It’s done using nginx’s "proxy_bind $remote_addr transparent;” directive.

I got it mostly working by making the upstream server send responses to the proxy:

iptables -t mangle -A PREROUTING -t mangle -j CONNMARK --set-mark 100
iptables -t mangle -A OUTPUT -t mangle -j CONNMARK --restore-mark
ip rule add fwmark 100 lookup 100
ip route add default via 192.168.1.1 table 100

And having the proxy capture these responses:

iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 --match multiport --sports 80,443 -j MARK --set-mark 100
ip rule add fwmark 100 lookup 100
ip route add local default dev lo table 100

Where the proxy has the IP 192.168.1.1 and upstream servers have IP range of 192.168.1.0/24. Only http and https packets are routed for testing purposes.

The reason for using CONNMARK, is because I want upstream servers to also be able to access the internet. Only responses should go through the proxy.

My strategy is to set connection mark when packets arrive from the proxy to the upstream, and copy that connection mark when upstream responds. When the upstream accesses the internet directly however, no connection mark should be set, because the packets are generated locally, they shouldn’t go through PREROUTING, and thus without connection mark copied.

This setup works perfectly when a client connects to the proxy.

But when I run "wget http://google.com” on the upstream server, the command never returns. According to tcpdump, the upstream first makes a 3-way tcp handshake, which behaves correctly, the proxy doesn’t see these packets. But once the upstream makes the actual http request. The proxy sees it, and also sees the continuing resending of PSH, ACK packets to google.

I have no idea how that could happen. That looks like the locally generated http requests have fmark on them. The proxy and the upstream live in docker containers (that joins the same created bridge network), which I’m not sure if is relevant.

Did I miss something or I should not use CONNMARK?

Regards
Glen

Re: Surprising CONNMARK behaviors

[Glen Huang]

Never mind. It didn’t work because internet responses also went through PREROUTING.

> On Sep 25, 2020, at 11:55 AM, Glen Huang <heyhgl@gmail.com> wrote:
> 
> Hi, all,
> 
> I have some CONNMARK rules that only partially behave as intended. I think either my understanding of how CONNMARK works is wrong or the packets I’m sending are idiosyncratic. Would really appreciate it if someone could shed some light.
> 
> I’m trying to create a transparent proxy using nginx that functions as a tcp load balancer. Whenever a client connects to the proxy, the proxy sends to/receives from the upstream server using client IP as the source/destination IP. So so that the client IP shows up in upstream logs instead of the proxy’s. It’s done using nginx’s "proxy_bind $remote_addr transparent;” directive.
> 
> I got it mostly working by making the upstream server send responses to the proxy:
> 
> iptables -t mangle -A PREROUTING -t mangle -j CONNMARK --set-mark 100
> iptables -t mangle -A OUTPUT -t mangle -j CONNMARK --restore-mark
> ip rule add fwmark 100 lookup 100
> ip route add default via 192.168.1.1 table 100
> 
> And having the proxy capture these responses:
> 
> iptables -t mangle -A PREROUTING -p tcp -s 192.168.1.0/24 --match multiport --sports 80,443 -j MARK --set-mark 100
> ip rule add fwmark 100 lookup 100
> ip route add local default dev lo table 100
> 
> Where the proxy has the IP 192.168.1.1 and upstream servers have IP range of 192.168.1.0/24. Only http and https packets are routed for testing purposes.
> 
> The reason for using CONNMARK, is because I want upstream servers to also be able to access the internet. Only responses should go through the proxy.
> 
> My strategy is to set connection mark when packets arrive from the proxy to the upstream, and copy that connection mark when upstream responds. When the upstream accesses the internet directly however, no connection mark should be set, because the packets are generated locally, they shouldn’t go through PREROUTING, and thus without connection mark copied.
> 
> This setup works perfectly when a client connects to the proxy.
> 
> But when I run "wget http://google.com” on the upstream server, the command never returns. According to tcpdump, the upstream first makes a 3-way tcp handshake, which behaves correctly, the proxy doesn’t see these packets. But once the upstream makes the actual http request. The proxy sees it, and also sees the continuing resending of PSH, ACK packets to google.
> 
> I have no idea how that could happen. That looks like the locally generated http requests have fmark on them. The proxy and the upstream live in docker containers (that joins the same created bridge network), which I’m not sure if is relevant.
> 
> Did I miss something or I should not use CONNMARK?
> 
> Regards
> Glen