GPE traffic gets stuck between ppp and br0 interface, ipv4 traffic works fine, ideas?

GPE traffic gets stuck between ppp and br0 interface, ipv4 traffic works fine, ideas?

[Jelle de Jong]

[-- Attachment #1: Type: text/plain, Size: 2477 bytes --]

Hello everybody,

I am trying to add an GPE tunnel over a fully working IPv4 setup and I 
am having the issue that the GPE packages get stuck on the Debian 
modem/router at ppp0 (package visable) and the br0 (package gone)

I have setup two virtual machine both with an external ip address, 
without firewall and I have setup an standard GPE tunnel between the two 
for testing. One virtual machine is in a data centre the other one is at 
our office server room.

uplink fiber -> switch -> bond0 -> vlan6 -> ppp0 -> br0 -> vlan34 -> 
switch -> bond0 -> br0 -> eth0 (virtual machine).

I can ping both virtual machines on there external IP address just fine 
and I dont have any other issues with this setup and it has been stable. 
I wanted to add a GPE tunnel as I need to route extra IP address in the 
future.

So I got a ping running on both machines, but they do not get a reply back.

When looking with tcpdump I see both the incoming ICMP request (the one 
from the data centre, ends at ppp0 on our xs4all Debian modem (its uses 
ppp to get the external ip addr).

Then the ICMP request from our virtual machine nicely makes it to our 
bridge on our xs4all Debian modem, but then stops and does not go out 
our PPP interface with the external ip addr.

I have added full information in the attachment.

root@xs4all:~# tcpdump -i ppp0 proto 47 and ip[33]=0x01 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 
262144 bytes
18:43:50.046573 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 
10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1347, length 64

root@xs4all:~# tcpdump -i br0 proto 47 and ip[33]=0x01 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:44:07.293275 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 
10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 821, length 64

root@xs4all:~# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0

/sbin/iptables --append FORWARD --protocol GRE --in-interface br0 
--out-interface ppp0 -j ACCEPT
/sbin/iptables --append FORWARD --protocol GRE --in-interface ppp0 
--out-interface ppp0 -j ACCEPT

What am I missing? How come the GPE data is not routed like the ipv4 
data is? Any ideas how to fix my issue?

Much appreciated.

Kind regards,

Jelle de Jong

[-- Attachment #2: debug-gre-2021-10-13.txt --]
[-- Type: text/plain, Size: 11085 bytes --]

root@xs4all:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
4: enp3s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
6: vlan3@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
    inet 80.127.158.80/27 brd 80.127.158.95 scope global br0
       valid_lft forever preferred_lft forever
8: vlan6@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
9: vlan34@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 00:0d:b9:4a:56:28 brd ff:ff:ff:ff:ff:ff
12: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 3
    link/ppp
    inet 62.251.96.42 peer 194.109.5.175/32 scope global ppp0
       valid_lft forever preferred_lft forever

root@xs4all:~# ip route show
default dev ppp0 scope link
80.127.158.64/27 dev br0 proto kernel scope link src 80.127.158.80
194.109.5.175 dev ppp0 proto kernel scope link src 62.251.96.42


root@xs4all:~# brctl show br0
bridge name bridge id       STP enabled interfaces
br0     8000.000db94a5628   no      vlan3
                            vlan34

root@xs4all:~# ping 80.127.158.82
PING 80.127.158.82 (80.127.158.82) 56(84) bytes of data.
64 bytes from 80.127.158.82: icmp_seq=1 ttl=64 time=0.686 ms
64 bytes from 80.127.158.82: icmp_seq=2 ttl=64 time=0.631 ms
64 bytes from 80.127.158.82: icmp_seq=3 ttl=64 time=0.579 ms
64 bytes from 80.127.158.82: icmp_seq=4 ttl=64 time=0.509 ms

--- 80.127.158.82 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3052ms
rtt min/avg/max/mdev = 0.509/0.601/0.686/0.067 ms

root@xs4all:~# iptables-save
# Generated by iptables-save v1.6.0 on Wed Oct 13 18:40:28 2021
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:f2b-sshd - [0:0]
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A INPUT -m state --state INVALID -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p gre -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i bond0 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -p gre -j ACCEPT
-A FORWARD -i br0 -o ppp0 -p gre -j ACCEPT
-A FORWARD -i ppp0 -o ppp0 -p gre -j ACCEPT
-A FORWARD -i bond0 -o ppp0 -j ACCEPT
-A FORWARD -i ppp0 -o br0 -j ACCEPT
-A FORWARD -i br0 -o ppp0 -j ACCEPT
-A FORWARD -i br0 -o tun_extra_ip -j ACCEPT
-A FORWARD -i tun_extra_ip -o br0 -j ACCEPT
-A FORWARD -j LOG
-A FORWARD -j DROP
-A OUTPUT -m state --state INVALID -j DROP
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -j ACCEPT
-A OUTPUT -p udp -m udp --dport 53 -j ACCEPT
-A OUTPUT -p udp -m udp --sport 53 -j ACCEPT
-A OUTPUT -p udp -m udp --dport 123 -j ACCEPT
-A OUTPUT -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A OUTPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -p gre -j ACCEPT
-A OUTPUT -j LOG
-A OUTPUT -j DROP
-A f2b-sshd -s 220.168.85.68/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 168.121.104.115/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 124.43.9.184/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 121.4.95.102/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
COMMIT
# Completed on Wed Oct 13 18:40:28 2021
# Generated by iptables-save v1.6.0 on Wed Oct 13 18:40:28 2021
*nat
:PREROUTING ACCEPT [214:18637]
:INPUT ACCEPT [11:732]
:OUTPUT ACCEPT [2:152]
:POSTROUTING ACCEPT [202:17913]
-A PREROUTING -d 62.251.96.42/32 -i ppp0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 80.127.158.83:2231
COMMIT
# Completed on Wed Oct 13 18:40:28 2021
# Generated by iptables-save v1.6.0 on Wed Oct 13 18:40:28 2021
*mangle
:PREROUTING ACCEPT [182698:22341160]
:INPUT ACCEPT [341:25220]
:FORWARD ACCEPT [182357:22315940]
:OUTPUT ACCEPT [299:43855]
:POSTROUTING ACCEPT [182495:22327720]
COMMIT
# Completed on Wed Oct 13 18:40:28 2021


root@xs4all:~# tcpdump -i br0 proto GRE && proto ICMP -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:30:33.210991 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 26, length 64
18:30:34.234951 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 27, length 64
18:30:35.258997 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 28, length 64
18:30:36.283019 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 29, length 64
18:30:37.306992 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 30, length 64
18:30:38.331035 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 31, length 64
18:30:39.355031 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 32, length 64
18:30:40.379033 IP a80-127-158-82.adsl.xs4all.nl > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 33, length 64

root@xs4all:~# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0

/sbin/iptables --append FORWARD --protocol GRE --in-interface br0 --out-interface ppp0 -j ACCEPT
/sbin/iptables --append FORWARD --protocol GRE --in-interface ppp0 --out-interface ppp0 -j ACCEPT

root@xs4all:~# tcpdump -i ppp0 proto 47 and ip[33]=0x01 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
18:43:50.046573 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1347, length 64
18:43:51.070586 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1348, length 64
18:43:52.094562 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1349, length 64
18:43:53.118594 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1350, length 64
18:43:54.142591 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1351, length 64
18:43:55.166542 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1352, length 64
18:43:56.190601 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1353, length 64
18:43:57.214594 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1354, length 64
18:43:58.238831 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1355, length 64
18:43:59.262578 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1356, length 64
18:44:00.286560 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1357, length 64
18:44:01.310600 IP 185.87.185.190 > 80.127.158.82: GREv0, length 88: IP 10.0.0.1 > 10.0.0.2: ICMP echo request, id 5137, seq 1358, length 64
^C
12 packets captured
16 packets received by filter
0 packets dropped by kernel
root@xs4all:~# tcpdump -i br0 proto 47 and ip[33]=0x01 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:44:07.293275 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 821, length 64
18:44:08.317276 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 822, length 64
18:44:09.341296 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 823, length 64
18:44:10.365296 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 824, length 64
18:44:11.389329 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 825, length 64
18:44:12.413425 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 826, length 64
18:44:13.437331 IP 80.127.158.82 > 185.87.185.190: GREv0, length 88: IP 10.0.0.2 > 10.0.0.1: ICMP echo request, id 31130, seq 827, length 64
^C
7 packets captured
15 packets received by filter
0 packets dropped by kernel


A
IP: 185.87.185.190
GRE tunnel internal IP: 10.0.0.1

B
IP: 80.127.158.82
GRE tunnel internal IP: 10.0.0.2

A & B
echo net.ipv4.ip_forward = 1 | tee /etc/sysctl.d/06iptables.conf
sysctl -p /etc/sysctl.d/06iptables.conf
sysctl -a | grep ip_forward

rmmod ip_gre
rmmod nf_conntrack_proto_gre

modprobe ip_gre
modprobe nf_conntrack_proto_gre

A
sudo ip tunnel add gre1 mode gre local 185.87.185.190 remote 80.127.158.82 ttl 255
sudo ip link set gre1 up
sudo ip addr add 10.0.0.1 dev gre1
sudo ip route add 10.0.0.0/24 dev gre1

B
sudo ip tunnel add gre1 mode gre local 80.127.158.82 remote 185.87.185.190 ttl 255
sudo ip link set gre1 up
sudo ip addr add 10.0.0.2 dev gre1
sudo ip route add 10.0.0.0/24 dev gre1
# sudo ip addr add 10.0.0.2/30 dev gre1

B
ping 10.0.0.1

A
ping 10.0.0.2