Packet disappears after DNAT?

Packet disappears after DNAT?

[Scott Bronson]

Is there any reason for a packet to just disappear when it gets
DNATed?  (other than rp_filter, which is disabled)


Background:

I have a typical libvirt setup, where VMs share a private network,
and the host masquerades them onto the internet.

In addition, the host is forwarding port 25 to a VM using DNAT.


The Problem:

I log into the VM and try to connect to port 25 on the host's public
IP.

  $ ncat u32.net 25

On the VM, that command just times out.  It works from the internet
and on the host (thanks Noel).

Logging demonstrates that the VM sends the packet to the host just fine.
Then, after the host DNATs it, it just disappears.  I would expect it to
get routed back to the VM.


More Details:

The VM creates the packet: src=192.168.122.10:23456 dst=173.233.67.174:25

  * it arrives on the host's raw:PREROUTING
  * it moves to mangle:PREROUTING
  * it then goes to nat:PREROUTING
    * Rule 5 DNATs the destination to 192.168.122.10

And that's it!  Nothing more happens, the packet is gone.


Here's my research with tables and traces:
  https://gist.github.com/bronson/c857a462edb0c6eeab2d

Can anyone tell me what to look at here?  I'm stumped.

    - Scott

Re: Packet disappears after DNAT?

[Anton Danilov]

Hello.
Have you enabled the ip forward?

2015-12-04 12:43 GMT+03:00 Scott Bronson <bronson@rinspin.com>:
> Is there any reason for a packet to just disappear when it gets
> DNATed?  (other than rp_filter, which is disabled)
>
>
> Background:
>
> I have a typical libvirt setup, where VMs share a private network,
> and the host masquerades them onto the internet.
>
> In addition, the host is forwarding port 25 to a VM using DNAT.
>
>
> The Problem:
>
> I log into the VM and try to connect to port 25 on the host's public
> IP.
>
>   $ ncat u32.net 25
>
> On the VM, that command just times out.  It works from the internet
> and on the host (thanks Noel).
>
> Logging demonstrates that the VM sends the packet to the host just fine.
> Then, after the host DNATs it, it just disappears.  I would expect it to
> get routed back to the VM.
>
>
> More Details:
>
> The VM creates the packet: src=192.168.122.10:23456 dst=173.233.67.174:25
>
>   * it arrives on the host's raw:PREROUTING
>   * it moves to mangle:PREROUTING
>   * it then goes to nat:PREROUTING
>     * Rule 5 DNATs the destination to 192.168.122.10
>
> And that's it!  Nothing more happens, the packet is gone.
>
>
> Here's my research with tables and traces:
>   https://gist.github.com/bronson/c857a462edb0c6eeab2d
>
> Can anyone tell me what to look at here?  I'm stumped.
>
>     - Scott
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Anton.

Re: Packet disappears after DNAT?

[Scott Bronson]

On Fri, Dec 4, 2015 at 3:55 AM, Anton Danilov
<littlesmilingcloud@gmail.com> wrote:
> Have you enabled the ip forward?

Yep, alas.  Connections from the host and the public IP work great.
It's just connections from the VM where the packet disappears.


I also should have mentioned that this is on Centos 7, and:

[root@ex ~]# cat /proc/version
Linux version 3.10.0-229.20.1.el7.x86_64
(builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat
4.8.3-9) (GCC) ) #1 SMP Tue Nov 3 19:10:07 UTC 2015

[root@ex ~]# iptables --version
iptables v1.4.21

Re: Packet disappears after DNAT?

[Pascal Hambourg]

Scott Bronson a écrit :
> Is there any reason for a packet to just disappear when it gets
> DNATed?  (other than rp_filter, which is disabled)
> 
> The VM creates the packet: src=192.168.122.10:23456 dst=173.233.67.174:25
> 
>   * it arrives on the host's raw:PREROUTING
>   * it moves to mangle:PREROUTING
>   * it then goes to nat:PREROUTING
>     * Rule 5 DNATs the destination to 192.168.122.10
> 
> And that's it!  Nothing more happens, the packet is gone.
> 
> Here's my research with tables and traces:
>   https://gist.github.com/bronson/c857a462edb0c6eeab2d

IIUC, you're looping back traffic on a bridge interface. Bridge-nf is
enabled by default and can cause weird behaviour with NAT. Try to
disable it :

echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

Without that, after DNAT, the packet is not routed but bridged. But I
suspect that the bridge code won't forward it back to its original port
due to its source MAC address.

Re: Packet disappears after DNAT?

[Anton Danilov]

show the output of 'ip route get 192.168.122.10 from 192.168.122.10
iif virbr0' command

2015-12-04 19:18 GMT+03:00 Scott Bronson <bronson@rinspin.com>:
> On Fri, Dec 4, 2015 at 3:55 AM, Anton Danilov
> <littlesmilingcloud@gmail.com> wrote:
>> Have you enabled the ip forward?
>
> Yep, alas.  Connections from the host and the public IP work great.
> It's just connections from the VM where the packet disappears.
>
>
> I also should have mentioned that this is on Centos 7, and:
>
> [root@ex ~]# cat /proc/version
> Linux version 3.10.0-229.20.1.el7.x86_64
> (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat
> 4.8.3-9) (GCC) ) #1 SMP Tue Nov 3 19:10:07 UTC 2015
>
> [root@ex ~]# iptables --version
> iptables v1.4.21



-- 
Anton.

Re: Packet disappears after DNAT?

[Scott Bronson]

On Sun, Dec 6, 2015 at 12:49 AM, Anton Danilov
<littlesmilingcloud@gmail.com> wrote:
> show the output of 'ip route get 192.168.122.10 from 192.168.122.10
> iif virbr0' command

[root@ex ~]# ip route get 192.168.122.10 from 192.168.122.10 iif virbr0
192.168.122.10 from 192.168.122.10 dev virbr0
    cache <redirect>  iif *

Re: Packet disappears after DNAT?

[Scott Bronson]

On Sat, Dec 5, 2015 at 2:31 AM, Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
> IIUC, you're looping back traffic on a bridge interface.

That sounds right.  The packet arrives on virbr0, and I want to send it
right back where it came from.

> Bridge-nf is
> enabled by default and can cause weird behaviour with NAT. Try to
> disable it :
>
> echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
>
> Without that, after DNAT, the packet is not routed but bridged. But I
> suspect that the bridge code won't forward it back to its original port
> due to its source MAC address.

It still times out.  Is it worth installing the logging to see if the
packet is still disappearing at the DNAT step?  Guessing no.

Maybe I can DNAT the source MAC too?  :)

Re: Packet disappears after DNAT?

[Pascal Hambourg]

Scott Bronson a écrit :
> On Sat, Dec 5, 2015 at 2:31 AM, Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
>> IIUC, you're looping back traffic on a bridge interface.
> 
> That sounds right.  The packet arrives on virbr0, and I want to send it
> right back where it came from.
> 
>> Bridge-nf is
>> enabled by default and can cause weird behaviour with NAT. Try to
>> disable it :
>>
>> echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
>>
>> Without that, after DNAT, the packet is not routed but bridged. But I
>> suspect that the bridge code won't forward it back to its original port
>> due to its source MAC address.
> 
> It still times out.

You must also MASQUERADE or SNAT these packets, otherwise the reply
packets won't be sent back to the host and be de-NATed properly. AFAICS,
your current ruleset avoids doing it :

> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE

> Is it worth installing the logging to see if the
> packet is still disappearing at the DNAT step?

Sure. Or at least do a packet capture on the interface.

> Maybe I can DNAT the source MAC too?  :)

I guess you may SNAT the source MAC address. It might do the trick

Re: Packet disappears after DNAT?

[Pascal Hambourg]

Pascal Hambourg a écrit :
> Scott Bronson a écrit :
> 
>> Maybe I can DNAT the source MAC too?  :)
> 
> I guess you may SNAT the source MAC address. It might do the trick

Oops I sent too fast.
I do not think that ebtable can SNAT before the bridging decision.

Re: Packet disappears after DNAT?

[Scott Bronson]

On Sat, Dec 5, 2015 at 2:31 AM, Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
> Bridge-nf is
> enabled by default and can cause weird behaviour with NAT. Try to
> disable it :
>
> echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

Yes, thank you!  I never would have found that.  Without it, the
VM->Host->VM NAT
just eats the packets.


On Mon, Dec 7, 2015 at 12:01 AM, Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
> You must also MASQUERADE or SNAT these packets, otherwise the reply
> packets won't be sent back to the host and be de-NATed properly

You're absolutely right.  With two more rules:

iptables -t nat -I POSTROUTING -s 192.168.122.10 -d 192.168.122.10 -p
tcp -j MASQUERADE
iptables -t nat -I POSTROUTING -s 192.168.122.10 -d 192.168.122.10 -p
udp -j MASQUERADE

it works!


Gotta say, unning `echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables` in my
script makes me a little queasy...  but this page suggests that no other option
will work cross-distro:

     http://wiki.libvirt.org/page/Net.bridge-nf-call_and_sysctl.conf

I guess that's still the case?


Huge thanks Pascal.

    - Scott

Re: Packet disappears after DNAT?

[Pablo Neira Ayuso]

On Tue, Dec 15, 2015 at 04:21:30AM -0800, Scott Bronson wrote:
> On Sat, Dec 5, 2015 at 2:31 AM, Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
> > Bridge-nf is
> > enabled by default and can cause weird behaviour with NAT. Try to
> > disable it :
> >
> > echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
> 
> Yes, thank you!  I never would have found that.  Without it, the
> VM->Host->VM NAT
> just eats the packets.
> 
> 
> On Mon, Dec 7, 2015 at 12:01 AM, Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
> > You must also MASQUERADE or SNAT these packets, otherwise the reply
> > packets won't be sent back to the host and be de-NATed properly
> 
> You're absolutely right.  With two more rules:
> 
> iptables -t nat -I POSTROUTING -s 192.168.122.10 -d 192.168.122.10 -p
> tcp -j MASQUERADE
> iptables -t nat -I POSTROUTING -s 192.168.122.10 -d 192.168.122.10 -p
> udp -j MASQUERADE
> 
> it works!
> 
> 
> Gotta say, unning `echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables` in my
> script makes me a little queasy...  but this page suggests that no other option
> will work cross-distro:
> 
>      http://wiki.libvirt.org/page/Net.bridge-nf-call_and_sysctl.conf

JFYI: The br-nf thing was moved to a separated module since 3.18. So
now this finally requires explicit modprobing.

Re: Packet disappears after DNAT?

[Pascal Hambourg]

Scott Bronson a écrit :
> 
> iptables -t nat -I POSTROUTING -s 192.168.122.10 -d 192.168.122.10 -p
> tcp -j MASQUERADE
> iptables -t nat -I POSTROUTING -s 192.168.122.10 -d 192.168.122.10 -p
> udp -j MASQUERADE

Why restrict the MASQUERADE to TCP and UDP ?

> Gotta say, unning `echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables` in my
> script makes me a little queasy...  but this page suggests that no other option
> will work cross-distro:
> 
>      http://wiki.libvirt.org/page/Net.bridge-nf-call_and_sysctl.conf
> 
> I guess that's still the case?

Well, the problem is to make sure that the command is run after the
bridge module has been loaded. Maybe a way to achieve this is to use the
"install" command in /etc/modprobe.conf or /etc/modprobe.d/bridge.conf
to have modprobe itself run the command after loading the module :

install bridge /sbin/modprobe --ignore-install bridge $CMDLINE_OPTS ;
echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

(all in one line)

Of course it won't work if the module is loaded with insmod. Also, the
manpage states that this command may become deprecated.

Re: Packet disappears after DNAT?

[Pascal Hambourg]

Pablo Neira Ayuso a écrit :
> On Tue, Dec 15, 2015 at 04:21:30AM -0800, Scott Bronson wrote:
>>
>> Gotta say, unning `echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables` in my
>> script makes me a little queasy...  but this page suggests that no other option
>> will work cross-distro:
>>
>>      http://wiki.libvirt.org/page/Net.bridge-nf-call_and_sysctl.conf
> 
> JFYI: The br-nf thing was moved to a separated module since 3.18. So
> now this finally requires explicit modprobing.

I didn't know about this change, thanks. Being rather conservative, my
most recent kernel is still older than 3.18.

For the record, the new module name is br_netfilter.

Re: Packet disappears after DNAT?

[Scott Bronson]

On Tue, Dec 15, 2015 at 5:06 AM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On Tue, Dec 15, 2015 at 04:21:30AM -0800, Scott Bronson wrote:
>> Gotta say, running `echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables` in my
>> script makes me a little queasy.
>
> JFYI: The br-nf thing was moved to a separated module since 3.18. So
> now this finally requires explicit modprobing.

Thanks Pablo.  Is this the right logic to use for all kernels?

- if /proc/sys/net/bridge/bridge-nf-call-iptables doesn't exist
  - modprobe br_netfilter
- echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

    - Scott

Re: Packet disappears after DNAT?

[Scott Bronson]

On Tue, Dec 15, 2015 at 11:45 AM, Pascal Hambourg
<pascal@plouf.fr.eu.org> wrote:
> Scott Bronson a écrit :
>> iptables -t nat -I POSTROUTING -s 192.168.122.10 -d 192.168.122.10 -p
>> tcp -j MASQUERADE
>
> Why restrict the MASQUERADE to TCP and UDP ?

Good question.  Editing mistake.  I'm actually forwarding different
ports to different guests depending on port number:

iptables -t nat -A POSTROUTING -s 192.168.122.10/32 -d
192.168.122.10/32 -p udp --dport 53 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 192.168.122.10/32 -d
192.168.122.10/32 -p tcp --dport 53 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 192.168.122.12/32 -d
192.168.122.12/32 -p tcp --dport 25 -j MASQUERADE


FWIW, here's the script as it stands today:
    https://github.com/bronson/libvirt-hook-qemu/tree/hostfix
    https://github.com/bronson/libvirt-hook-qemu/blob/hostfix/qemu
And an example of the rules it inserts:
    https://github.com/bronson/libvirt-hook-qemu/blob/hostfix/test_qemu.py#L58-L73

It seems a little overcomplex but it's working great.  Huge thanks to everyone.

    - Scott

Re: Packet disappears after DNAT?

[Pascal Hambourg]

Scott Bronson a écrit :
> On Tue, Dec 15, 2015 at 5:06 AM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>>
>> JFYI: The br-nf thing was moved to a separated module since 3.18. So
>> now this finally requires explicit modprobing.
> 
> Thanks Pablo.  Is this the right logic to use for all kernels?
> 
> - if /proc/sys/net/bridge/bridge-nf-call-iptables doesn't exist
>   - modprobe br_netfilter
> - echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

Why load br_netfilter if it is not needed ?

I would do the following :

modprobe bridge
if /proc/sys/net/bridge/bridge-nf-call-iptables exists
  echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

Re: Packet disappears after DNAT?

[Pascal Hambourg]

Scott Bronson a écrit :
> 
> Good question.  Editing mistake.  I'm actually forwarding different
> ports to different guests depending on port number:
> 
> iptables -t nat -A POSTROUTING -s 192.168.122.10/32 -d
> 192.168.122.10/32 -p udp --dport 53 -j MASQUERADE
> iptables -t nat -A POSTROUTING -s 192.168.122.10/32 -d
> 192.168.122.10/32 -p tcp --dport 53 -j MASQUERADE
> iptables -t nat -A POSTROUTING -s 192.168.122.12/32 -d
> 192.168.122.12/32 -p tcp --dport 25 -j MASQUERADE

I don't think you need a separate rule for each forwarded port. One
global rule would do the job, as only forwarded packets would match the
source/destination address.


iptables -t nat -A POSTROUTING -s 192.168.122.10/32 -d 192.168.122.10/32
  -j MASQUERADE