Bridging behavior apparently changed around the Fedora 14 time

Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

I ran into a strange situation - I am using a firewall set up as a
bridge.  Physical device eth1 is the private LAN side, eth0 is the
public Internet side.  I setup bridge br0 to bridge eth0 and eth1
together.  I need a bridge because this site has a couple of nodes on
the LAN side that need real public IP Addresses.  

This site also has a few web and ftp sites.  These are NATed behind the
firewall, but internal users need to see them the same way as the rest
of the world.  So I use some iptables SNAT and DNAT rules to make this
happen.  Device br0 has the relevant public IP Address(es) and then NATs
to the appropriate private IP Address(es).  The ruleset works and the
system has been up and running for several years.  

I recently replaced the old system with a new one running Fedora 14 and
that's when the weird behavior started.  

Now, when internal people try to look at those web/ftp sites using the
public IP Addresses, they get nowhere.  Unless I watch with tcpdump -
and then while I'm watching , all works as it should.  With some help,
we figured out the reason it works when watching with tcpdump - because
tcpdump puts the device being monitored into promiscuous mode.  

And, sure enough, when I do:
    ip link set br0 promisc on

everything works as it should.

Looking at "ip link show", it looks like bridge br0 takes on the MAC
address of physical NIC eth0.  But the internal LAN is connected to
physical eth1.  I wonder if this behavior is different than the older
version?  If the MAC Address for bridge br0 is different than the
physical device I'm actually connected to, I wonder if bridging "thinks"
I'm trying to hit a foreign MAC Address - especially since I'm doing
both SNAT and DNAT on the same packet?  

[root@ehac-fw2011 ~]# ip link show eth1
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
    link/ether 00:0d:88:31:d8:24 brd ff:ff:ff:ff:ff:ff
[root@ehac-fw2011 ~]# ip link show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
    link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff
[root@ehac-fw2011 ~]#
[root@ehac-fw2011 ~]# ip link show br0
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc prio state
UNKNOWN
    link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff

[root@ehac-fw2011 ~]# brctl show macs br0
bridge name     bridge id               STP enabled     interfaces
br0             8000.0003473a5979       no              eth0
                                                        eth1
Hmmmm - so a packet comes in on eth1, with a destination MAC Address
belonging to physical eth0.  So eth1 throws it away because it "thinks"
this is a foreign MAC Address?  But this all worked before, so what's
different?  Or were earlier bridges in promiscuous mode by default and
now they're not?  Have I stumbled across a new bridging bug?  Is the
best workaround to just put br0 into promiscuous mode?

Thanks

- Greg Scott

Re: Bridging behavior apparently changed around the Fedora 14 time

[Stephen Hemminger]

On Mon, 11 Jul 2011 13:25:46 -0500
"Greg Scott" <GregScott@Infrasupport.com> wrote:

> I ran into a strange situation - I am using a firewall set up as a
> bridge.  Physical device eth1 is the private LAN side, eth0 is the
> public Internet side.  I setup bridge br0 to bridge eth0 and eth1
> together.  I need a bridge because this site has a couple of nodes on
> the LAN side that need real public IP Addresses.  
> 
> This site also has a few web and ftp sites.  These are NATed behind the
> firewall, but internal users need to see them the same way as the rest
> of the world.  So I use some iptables SNAT and DNAT rules to make this
> happen.  Device br0 has the relevant public IP Address(es) and then NATs
> to the appropriate private IP Address(es).  The ruleset works and the
> system has been up and running for several years.  
> 
> I recently replaced the old system with a new one running Fedora 14 and
> that's when the weird behavior started.  
> 
> Now, when internal people try to look at those web/ftp sites using the
> public IP Addresses, they get nowhere.  Unless I watch with tcpdump -
> and then while I'm watching , all works as it should.  With some help,
> we figured out the reason it works when watching with tcpdump - because
> tcpdump puts the device being monitored into promiscuous mode.  
> 
> And, sure enough, when I do:
>     ip link set br0 promisc on
> 
> everything works as it should.

Please provide more configuration information like:
  - NIC type
  - iptables and bridge and address configuration

Do you have reverse path filtering enabled/disabled?

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> Please provide more configuration information like:
>  - NIC type
>  - iptables and bridge and address configuration

The NICs are 3Com 3C905B's.  The iptables config is complex - I'll paste
in the relevant pieces.  Note that it all works as expected when I put
the br0 device into promiscuous mode.  I included all the layer 2 MAC
address info in the original email.  

> Do you have reverse path filtering enabled/disabled?

rp_filter doesn't seem to effect the behavior one way or the other.
I've tried with rp_filter set to both 0 and 1 on eth0, eth1, and br0
with no change in behavior.  

Here are the relevant parts of my rc.firewall script:

.
.
.
echo "Setting up aliases for public IP addresses"

/sbin/ifconfig ${BR_IFACE}:0 $PUBLIC_GRE_IP netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
#/sbin/ifconfig ${BR_IFACE}:1 $PUBLIC_EMAIL_IP netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
# EMAIL and GRE share the same IP Address
/sbin/ifconfig ${BR_IFACE}:2 ${PUBLIC_FTP_IP[0]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
/sbin/ifconfig ${BR_IFACE}:3 ${PUBLIC_FTP_IP[1]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
/sbin/ifconfig ${BR_IFACE}:4 ${PUBLIC_FTP_IP[2]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
/sbin/ifconfig ${BR_IFACE}:5 ${PUBLIC_FTP_IP[3]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
#/sbin/ifconfig ${BR_IFACE}:6 ${PUBLIC_VTC_IP[0]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
#/sbin/ifconfig ${BR_IFACE}:7 ${PUBLIC_VTC_IP[1]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
# The VTC devices are now bridged.  They have their own IP Addresses.
#/sbin/ifconfig ${BR_IFACE}:8 ${PUBLIC_WEB_IP[0]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
#/sbin/ifconfig ${BR_IFACE}:9 ${PUBLIC_WEB_IP[1]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
#/sbin/ifconfig ${BR_IFACE}:10 ${PUBLIC_WEB_IP[2]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
#/sbin/ifconfig ${BR_IFACE}:11 ${PUBLIC_WEB_IP[3]} netmask $INET_NETMASK
broadcast $INET_BCAST_ADDRESS
# Web and FTP sites have the same addresses.

echo "ARPinging the world so everyone knows our MAC Addresses"
/sbin/arping -A $INET_IP -c 1 -I $BR_IFACE
/sbin/arping -A $PUBLIC_GRE_IP -c 1 -I $BR_IFACE
for ADR in "${PUBLIC_FTP_IP[@]}"; do
    /sbin/arping -A $ADR -c 1 -I $BR_IFACE
done
for ADR in "${PUBLIC_WWW_IP[@]}"; do
    /sbin/arping -A $ADR -c 1 -I $BR_IFACE
done

/sbin/arping -A $TRUSTED1_IP -c 1 -I $BR_IFACE

/sbin/arping -A $DMZ_IP -c 1 -I $DMZ_IFACE
.
.
.
echo "	MASQUERADING requests to the internally hosted server apps"
echo "	(Email - SMTP, POP3, IMAP, OWA)"
$IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 25 -d $PRIVATE_EMAIL_IP -j MASQUERADE
$IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 110 -d $PRIVATE_EMAIL_IP -j MASQUERADE
$IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 143 -d $PRIVATE_EMAIL_IP -j MASQUERADE
$IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 80 -d $PRIVATE_EMAIL_IP -j MASQUERADE
$IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 443 -d $PRIVATE_EMAIL_IP -j MASQUERADE

echo "	(Internally hosted ftp sites)"
for ADR in "${PUBLIC_FTP_IP[@]}"; do
    $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 21 -d $ADR -j MASQUERADE
done

echo "	(Internally hosted web sites)"
for ADR in "${PUBLIC_WWW_IP[@]}"; do
    $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 80 -d $ADR -j MASQUERADE
    $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \
	-p TCP --dport 443 -d $ADR -j MASQUERADE
done
.
.
.
echo "	FTP sites (port 21 inbound)"
for ADR in "${PRIVATE_FTP_IP[@]}"; do
    $IPTABLES -A FORWARD -p TCP --dport 21 -s 0/0 -d $ADR -j allowed 
done

echo "	Web (HTTP) sites (port 80 inbound)"
for ADR in "${PRIVATE_WWW_IP[@]}"; do
    $IPTABLES -A FORWARD -p TCP --dport 80 -s 0/0 -d $ADR -j allowed 
done
.
.
.

echo "	FTP sites (TCP port 21)"
for ((i = 0 ; i < ${#PUBLIC_FTP_IP[@]} ; i++ ))
do
    $IPTABLES -t nat -A PREROUTING -d ${PUBLIC_FTP_IP[i]} \
	-p tcp --dport 21 -j DNAT --to ${PRIVATE_FTP_IP[i]}
	# -m mark --mark 1 (from the Internet) removed so we
	# DNAT for internal users also.
done

echo "	Web (HTTP) sites (TCP port 80)"
for ((i = 0 ; i < ${#PUBLIC_WWW_IP[@]} ; i++ ))
do
    $IPTABLES -t nat -A PREROUTING -d ${PUBLIC_WWW_IP[i]} \
	-p tcp --dport 80 -j DNAT --to ${PRIVATE_WWW_IP[i]}
	# -m mark --mark 1 (from the Internet) removed so we
	# DNAT for internal users also.
done

echo "  Email - SMTP, POP3, IMAP, OWA; provision for internal users"
$IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \
		-p tcp --dport 25 -j DNAT --to $PRIVATE_EMAIL_IP
$IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \
		-p tcp --dport 110 -j DNAT --to $PRIVATE_EMAIL_IP
$IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \
		-p tcp --dport 143 -j DNAT --to $PRIVATE_EMAIL_IP
$IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \
		-p tcp --dport 80 -j DNAT --to $PRIVATE_EMAIL_IP
$IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \
		-p tcp --dport 443 -j DNAT --to $PRIVATE_EMAIL_IP


As a benchmark test, from an internal host, I'll do telnet {public IP}
80.  And then listen in another window on the firewall with tcpdump.
Once we came across the promiscuous mode theory, we tried tcpdump with
-e and -p.  -e shows layer 2 stuff and -p does not put the device into
promiscuous mode.  Here is what we found:

For my testing, from a CMD window on the server hosting the website, I
do:

telnet aa.bb.115.151 80 (aa.bb obfuscated first 2 IP Address octets),
and then watch with tcpdump on the firewall.  

Also on the firewall, I did this:
echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter
echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo "0" > /proc/sys/net/ipv4/conf/br0/rp_filter

So that turned of any possible rp_filtering.  

And then on the firewall:
/usr/sbin/tcpdump -p -e -i br0 host aa.bb.115.151 -nn -vv

while the telnet was running in another window.

Nothing - no output, no matter what value I use for any of the rp_filter
files.  Nothing from my telnet session to port 80 hits bridge br0 on the
firewall.  

But here's the curious part - looking at physical interface eth1, I see
these packets when I do the same telnet test:

[root@ehac-fw2011 ~]# /usr/sbin/tcpdump -p -e -i eth1 host aa.bb.115.151
-nn -vv
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size
65535 bytes
14:51:33.412280 00:0f:20:f7:06:18 > 00:03:47:3a:59:79, ethertype IPv4
(0x0800), length 62: (tos 0x0, ttl 128, id 18631, offset 0, flags [DF],
proto TCP (6), length 48)
    192.168.10.2.54092 > aa.bb.115.151.80: Flags [S], cksum 0xddb2
(correct), seq 4146878900, win 65535, options [mss 1460,nop,nop,sackOK],
length 0
14:51:39.427928 00:0f:20:f7:06:18 > 00:03:47:3a:59:79, ethertype IPv4
(0x0800), length 62: (tos 0x0, ttl 128, id 18733, offset 0, flags [DF],
proto TCP (6), length 48)
    192.168.10.2.54092 > aa.bb.115.151.80: Flags [S], cksum 0xddb2
(correct), seq 4146878900, win 65535, options [mss 1460,nop,nop,sackOK],
length 0 ^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
[root@ehac-fw2011 ~]#


So for some reason, the bridging stuff is apparently not passing from
eth1 to br0.  The single, one and only change to make this work is:

ip link set br0 promisc on

When I do above, then the internal packets to external IP Addresses flow
out to the firewall and back in as expected.  

Did bridging used to put the bridge device into promiscuous mode by
default and now it doesn't?  Just trying to figure out what's different.


thanks

- Greg Scott

Re: Bridging behavior apparently changed around the Fedora 14 time

[Stephen Hemminger]

On Mon, 11 Jul 2011 15:41:35 -0500
"Greg Scott" <GregScott@Infrasupport.com> wrote:

> The NICs are 3Com 3C905B's.  The iptables config is complex - I'll paste
> in the relevant pieces.  Note that it all works as expected when I put
> the br0 device into promiscuous mode.  I included all the layer 2 MAC
> address info in the original email.  
> 
> > Do you have reverse path filtering enabled/disabled?  
> 
> rp_filter doesn't seem to effect the behavior one way or the other.
> I've tried with rp_filter set to both 0 and 1 on eth0, eth1, and br0
> with no change in behavior.  

What about console dmesg output.
Because you can fix by putting in promiscuous mode, I suspect the
problem is outside of the bridge itself, could be the drivers.
The bridge itself puts the device into promiscuous mode already.

Lastly, the Fedora and upstream kernels are not exactly the same.
Please retest with a standard upstream kernel (like 2.6.39.2).

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> What about console dmesg output.

I should probably turn off all firewall logging so I don't fill the ring
buffer with my log messages in, like, the first couple minutes after a
boot.  :)

> Please retest with a standard upstream kernel (like 2.6.39.2).

That's gonna take a while to put together a whole test environment with
the latest kernel.org kernel.

> The bridge itself puts the device into promiscuous mode already.

Uhmmmm - no it didn't.  Remember, I put br0 into promiscuous mode myself
by hand - take a look at this.  Note eth0 and eth1 are not in
promiscuous mode.  I wonder how it would behave if I put the physical
devices into promiscuous mode and left br0 alone?  This I can easily
test during off hours.  

[root@ehac-fw2011 gregs]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
DOWN qlen 1000
    link/ether 00:0e:7f:2d:d0:6e brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
    link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
    link/ether 00:0d:88:31:d8:24 brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc prio
state UNKNOWN
    link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff
[root@ehac-fw2011 gregs]#
  
- Greg

Re: Bridging behavior apparently changed around the Fedora 14 time

[Stephen Hemminger]

On Mon, 11 Jul 2011 16:08:14 -0500
"Greg Scott" <GregScott@Infrasupport.com> wrote:

> > What about console dmesg output.
> 
> I should probably turn off all firewall logging so I don't fill the ring
> buffer with my log messages in, like, the first couple minutes after a
> boot.  :)
> 
> > Please retest with a standard upstream kernel (like 2.6.39.2).
> 
> That's gonna take a while to put together a whole test environment with
> the latest kernel.org kernel.
> 
> > The bridge itself puts the device into promiscuous mode already.

The bridge code calls dev_set_promiscuity() which should
be changing device mode. But it could be that netdev core is 
resetting/changing/breaking that.

Re: Bridging behavior apparently changed around the Fedora 14 time

[Ben Greear]

On 07/11/2011 02:10 PM, Stephen Hemminger wrote:
> On Mon, 11 Jul 2011 16:08:14 -0500
> "Greg Scott"<GregScott@Infrasupport.com>  wrote:
>
>>> What about console dmesg output.
>>
>> I should probably turn off all firewall logging so I don't fill the ring
>> buffer with my log messages in, like, the first couple minutes after a
>> boot.  :)
>>
>>> Please retest with a standard upstream kernel (like 2.6.39.2).
>>
>> That's gonna take a while to put together a whole test environment with
>> the latest kernel.org kernel.
>>
>>> The bridge itself puts the device into promiscuous mode already.
>
> The bridge code calls dev_set_promiscuity() which should
> be changing device mode. But it could be that netdev core is
> resetting/changing/breaking that.

Last time I checked, 'ifconfig' and similar output didn't
show promisc when NIC was actually promisc, unless the user
specified the promisc-ness.

You can read /sys/class/net/dev/eth0/flags and
see if flag 0x100 is set..if so, it's promisc.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> The bridge code calls dev_set_promiscuity() which should
> be changing device mode. But it could be that netdev core is 
> resetting/changing/breaking that.

Is it supposed to change the physical ethnn devices or the br device?

Here is what I do to set up the bridging. I do it myself right in the
script so I can control all the details.

.
.
.
#
# Setup bridging
#

echo "Setting up bridge $BR_IFACE to bridge $INET_IFACE with
$TRUSTED1_IFACE"

$BRCTL addbr $BR_IFACE
$BRCTL addif $BR_IFACE $INET_IFACE
$BRCTL addif $BR_IFACE $TRUSTED1_IFACE

echo "  Adding $BR_IP_SLASH and $TRUSTED1_IP_SLASH IP Addresses to
$BR_IFACE"
/sbin/ip addr add $BR_IP_SLASH broadcast $BR_BCAST_ADDRESS dev $BR_IFACE
/sbin/ip addr add $TRUSTED1_IP_SLASH broadcast $TRUSTED1_BCAST_ADDRESS
dev $BR_IFACE
/sbin/ip link set $BR_IFACE up

echo "  Removing $INET_IP_SLASH and $TRUSTED1_IP_SLASH from $INET_IFACE
and $TRUSTED1_IFACE"
/sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE
/sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE
/sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE
/sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE

echo "  Putting $BR_IFACE into promiscuous mode"
# This fixes a bug forwarding packets bound for external IP Addresses
# from the private LAN.

ip link set $BR_IFACE promisc on

#
# Set up aliases for public IP addresses
#
.
.
.


- Greg

Re: Bridging behavior apparently changed around the Fedora 14 time

[Stephen Hemminger]

On Mon, 11 Jul 2011 16:16:40 -0500
"Greg Scott" <GregScott@Infrasupport.com> wrote:

> > The bridge code calls dev_set_promiscuity() which should
> > be changing device mode. But it could be that netdev core is 
> > resetting/changing/breaking that.
> 
> Is it supposed to change the physical ethnn devices or the br device?

The physical device ethnn.

> Here is what I do to set up the bridging. I do it myself right in the
> script so I can control all the details.
> 
> .
> .
> .
> #
> # Setup bridging
> #
> 
> echo "Setting up bridge $BR_IFACE to bridge $INET_IFACE with
> $TRUSTED1_IFACE"
> 
> $BRCTL addbr $BR_IFACE
> $BRCTL addif $BR_IFACE $INET_IFACE
> $BRCTL addif $BR_IFACE $TRUSTED1_IFACE
> 
> echo "  Adding $BR_IP_SLASH and $TRUSTED1_IP_SLASH IP Addresses to
> $BR_IFACE"
> /sbin/ip addr add $BR_IP_SLASH broadcast $BR_BCAST_ADDRESS dev $BR_IFACE
> /sbin/ip addr add $TRUSTED1_IP_SLASH broadcast $TRUSTED1_BCAST_ADDRESS
> dev $BR_IFACE
> /sbin/ip link set $BR_IFACE up
> 
> echo "  Removing $INET_IP_SLASH and $TRUSTED1_IP_SLASH from $INET_IFACE
> and $TRUSTED1_IFACE"
> /sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE
> /sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE
> /sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE
> /sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE
> 
> echo "  Putting $BR_IFACE into promiscuous mode"
> # This fixes a bug forwarding packets bound for external IP Addresses
> # from the private LAN.
> 
> ip link set $BR_IFACE promisc on
> 

What is supposed to happen is that the bridge adds all the interface
MAC addresses to the forwarding table as permanent entries. To show the
forwarding table:
  # brctl showmacs br0

port no	mac addr		is local?	ageing timer
  1	c6:eb:2a:0c:12:eb	yes		   0.00

Then when packet arrives with that mac address it is handed up to
netfilter, then if not firewalled, it goes on to the IP stack.

There maybe protections against packet going back out the same interface
that are getting tripped by all the rewriting.

Re: Bridging behavior apparently changed around the Fedora 14 time

[David Lamparter]

On Mon, Jul 11, 2011 at 04:08:14PM -0500, Greg Scott wrote:
> Uhmmmm - no it didn't.  Remember, I put br0 into promiscuous mode myself
> by hand - take a look at this.  Note eth0 and eth1 are not in
> promiscuous mode.  I wonder how it would behave if I put the physical
> devices into promiscuous mode and left br0 alone?  This I can easily
> test during off hours.  
> 
> [root@ehac-fw2011 gregs]# ip link show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state
> DOWN qlen 1000
>     link/ether 00:0e:7f:2d:d0:6e brd ff:ff:ff:ff:ff:ff
> 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
>     link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff
> 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
>     link/ether 00:0d:88:31:d8:24 brd ff:ff:ff:ff:ff:ff
> 5: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc prio
> state UNKNOWN
>     link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff
> [root@ehac-fw2011 gregs]#

All bridge port devices should really be in promiscuous mode; either
this is just not displayed or the code is broken somewhere. On my 2.6.38
port both "ip" and "ifconfig" correctly show the devices as PROMISC
without any userspace tool setting this, no idea if this changed
recently.


However, I must say that your bug report sounds more like a 
forwarding-back-to-source-device IP-level problem. If I understand
your setup correctly, you have:

(servers)
 |
 +--------[eth0 <-br0-> eth1]------- internet
 |
(clients)

and what isn't working is the firewall forwarding packets that it
received on eth0/br0 back out on eth0/br0 on the IP level including
NATting? Or did I misunderstand?


-David


P.S.: any reason why you are (a) not using proxyarp instead of the
bridge and (b) not using a VLAN to put those servers with public IPs
in? You have a bit of a Frankennet there :/

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> If I understand your setup correctly, you have:
> 
> (servers)
>  |
>  +--------[eth0 <-br0-> eth1]------- internet
>  |
>  (clients)

Close.  Here's a better ASCII art picture.  There aren't really any
internal-->external clients.

Internet        firewall    Private LAN
----------------+******+---------+-------+
              eth0     eth1     NATed    H.323
               Bridge br0      servers  devices

The H.323 devices work better if they have real, public IP Addresses.
I've done them with NAT, but H.323 just works better if the devices
"think" they're directly connected to the Internet.  All the servers are
all on the physical private LAN side, physically behind the firewall.  

> Why not proxy ARP?

I used to use proxy ARP until I got burned really badly with what proxy
ARP really does - the NIC answers ARP requests (in proxy) for everyone
and anyone that asks with its own MAC address.  Think about that - proxy
ARP impersonates everyone and anyone on the LAN to which it's connected.


I had one of these in a colo center and for several hours, my box
Proxy-ARPed everyone and anyone on that same public network.  I don't
even like to think about how many public webservers I unintentionally
messed with that day.  Oh yes - and to make matters worse, that customer
had an IP load balancer behind my box nobody told me about and proxy ARP
messed that up too.  The stupid load balancer wouldn't clear its ARP
cache and had to be rebooted - and that took down a major website and
pretty much blew my only chance to do business with this customer.  All
in all, not one of my better days.  I decided right then and there, no
more proxy ARP. 

Bridging turns out to be a much cleaner and more polite way to do it.
Don't believe all the forum comments about the wonders of proxy-ARP.  

> Why not use a VLAN?

Because I really don't need one.  Plus it doesn't matter anyway - the
firewall can act as a router on a stick to go between my H.323 devices
and private IP servers.  With or without VLANs makes no difference in
this case.  

> You have a bit of a Frankennet there

I don't think so.  I have a single LAN with a couple of devices that
need public IP Addresses.  This isn't that unusual.  I have lots of
other sites doing it this way.  

> I must say that your bug report sounds more like a 
> forwarding-back-to-source-device IP-level problem.

I don't think it's an IP level problem.  I think it's a layer 2 problem
- and now I think the problem is, bridging is supposed to turn on
PROMISC mode and it didn't.  I had to do it by hand myself.  I never
paid attention to whether or not PROMISC mode was turned on with any of
the other ones like this I've done and never had a problem with it until
this one.  And as soon as I turned on PROMISC mode by hand, everything
worked as it should. If it was an IP problem, or routing problem, or
ruleset/filtering problem, why would PROMISC mode make any difference
one way or the other?

What I don't know yet is, is this a Fedora bug or a stock kernel bug?
Is anyone from Red Hat following this email list?

I think I will take a look at a few of my other bridged sites running
earlier versions and see if they turn on PROMISC mode on their bridged
NICs.  

- Greg

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> You can read /sys/class/net/dev/eth0/flags and
> see if flag 0x100 is set..if so, it's promisc.

Hmmm...

According to this, sure enough, all the relevant devices **are** set to
PROMISC, even though ip link show doesn't say so.

[root@ehac-fw2011 firewall-scripts]# more /sys/class/net/eth0/flags
0x1103
[root@ehac-fw2011 firewall-scripts]# more /sys/class/net/eth1/flags
0x1103
[root@ehac-fw2011 firewall-scripts]# more /sys/class/net/br0/flags
0x1103
[root@ehac-fw2011 firewall-scripts]#
[root@ehac-fw2011 firewall-scripts]#
[root@ehac-fw2011 firewall-scripts]# # eth2 is not part of the br0
bridge.
[root@ehac-fw2011 firewall-scripts]# more /sys/class/net/eth2/flags
0x1003
[root@ehac-fw2011 firewall-scripts]#

So the plot thickens - although I set br0 to PROMISC mode by hand here.

Gathering similar data from an older and larger bridged site, this one
running Fedora 9.  Here I did not set br0 to PROMISC by hand, but it's
set that way anyway. 

[root@lme-fw2 ~]#
[root@lme-fw2 ~]# brctl showmacs br0
port no mac addr                is local?       ageing timer
  2     00:00:aa:a9:c3:91       no                18.06
  2     00:00:aa:ab:25:5b       no               267.93
  2     00:01:6c:68:9b:fc       no                91.39
  2     00:01:6c:68:9c:15       no                19.81
  2     00:09:6b:83:30:fc       no                48.57
  2     00:0d:60:2b:cd:6b       no                62.93
  2     00:0d:60:70:bc:87       no               104.11
  2     00:0e:0c:ee:5f:f0       no                 1.27
  1     00:11:21:bd:7f:80       no                 0.03
  2     00:11:25:6b:3a:8b       no                39.18
  2     00:12:79:7f:f5:2f       no                 8.66
  2     00:14:38:de:4c:7a       no                 9.76
  2     00:15:b7:22:43:ff       no                 1.30
  2     00:15:b7:28:eb:fc       no               232.10
  2     00:15:b7:bf:bd:fc       no                52.28
  2     00:15:b7:c1:d8:fd       no               118.31
  2     00:15:b7:db:b0:fc       no               275.73
  2     00:15:f2:ea:93:c4       no                15.83
  2     00:16:17:e0:24:72       no               158.90
  2     00:17:31:46:e1:12       no                 7.74
  2     00:17:c5:14:e6:58       no                12.14
  2     00:17:c5:14:e6:59       no                 0.01
  2     00:19:21:07:8e:8c       no                37.98
  2     00:19:db:a1:56:3e       no                26.43
  2     00:1b:78:22:63:0c       no                19.87
  2     00:1c:25:97:70:2e       no                 9.63
  2     00:1c:c4:8e:dd:3e       no                12.90
  2     00:1e:0b:3e:7c:41       no                44.20
  2     00:1e:0b:3e:7f:11       no                 0.20
  2     00:1e:4f:b9:2f:00       no                14.06
  2     00:1f:29:14:f0:53       no                14.12
  2     00:21:5e:89:92:d0       no                15.30
  2     00:21:5e:b3:03:c1       no                 0.17
  2     00:22:68:59:b1:bf       no                36.45
  2     00:22:68:59:f5:a8       no                23.25
  2     00:22:68:59:f7:0a       no               106.53
  2     00:22:68:59:fe:24       no                 2.63
  2     00:24:01:07:15:a6       no                 2.12
  2     00:26:5a:7b:0f:e3       no               161.61
  2     00:30:80:d3:cd:20       no                 1.54
  2     00:40:af:81:7a:1b       no                15.81
  2     00:80:64:5e:5b:1b       no                66.77
  2     00:80:64:5e:5b:67       no                 1.67
  2     00:80:64:62:64:e8       no                 2.12
  2     00:90:dc:02:b5:9b       no                70.30
  2     00:90:dc:a4:0f:3b       no                 4.07
  2     00:90:dc:a4:0f:3c       no                22.30
  2     00:a0:c8:20:54:05       no                 0.23
  2     00:c0:b6:0a:da:3c       no                44.77
  2     00:c0:b6:0a:da:3d       no                44.77
  2     10:78:d2:a8:e6:84       no                34.50
  3     12:34:56:00:10:01       yes                0.00
  2     12:34:56:00:10:11       no                 1.02
  2     12:34:56:00:30:05       yes                0.00
  1     12:34:56:00:32:51       no                 1.02
  1     12:34:56:24:80:50       yes                0.00
  2     40:61:86:7e:8c:93       no                10.02
  2     6c:62:6d:5a:af:09       no                58.99
  2     70:71:bc:51:8e:73       no                18.06
  2     90:fb:a6:22:0a:06       no                23.24
  2     e4:1f:13:2d:63:a8       no                 0.31
[root@lme-fw2 ~]#
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.123456001001       no              eth0
                                                        eth1
                                                        eth2
pan0            8000.000000000000       no
[root@lme-fw2 ~]#  more /sys/class/net/eth2/flags
0x1103
[root@lme-fw2 ~]#  more /sys/class/net/eth1/flags
0x1103
[root@lme-fw2 ~]#  more /sys/class/net/eth0/flags
0x1103
[root@lme-fw2 ~]#  more /sys/class/net/br0/flags
0x1103
[root@lme-fw2 ~]#
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# ip link show dev eth2
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast
qlen 1000
    link/ether 12:34:56:00:10:01 brd ff:ff:ff:ff:ff:ff
[root@lme-fw2 ~]# ip link show dev eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
    link/ether 12:34:56:00:30:05 brd ff:ff:ff:ff:ff:ff
[root@lme-fw2 ~]# ip link show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen 1000
    link/ether 12:34:56:24:80:50 brd ff:ff:ff:ff:ff:ff
[root@lme-fw2 ~]# ip 
[root@lme-fw2 ~]# more /proc/version
Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0
20080428 (Red H
at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008
[link show dev br0
8: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether 12:34:56:00:10:01 brd ff:ff:ff:ff:ff:ff
[root@lme-fw2 ~]#

Re: Bridging behavior apparently changed around the Fedora 14 time

[David Lamparter]

Hi --

First of all, I still can't find your kernel version in any of your
mails. Can you please repeat the uname -a output of the affected box?

Anyway,

On Mon, Jul 11, 2011 at 09:38:51PM -0500, Greg Scott wrote:
> > Why not proxy ARP?
> 
> I used to use proxy ARP until I got burned really badly with what proxy
> ARP really does - the NIC answers ARP requests (in proxy) for everyone
> and anyone that asks with its own MAC address.  Think about that - proxy
> ARP impersonates everyone and anyone on the LAN to which it's connected.

Yes. That's why you use
	ip neighbor add proxy 1.2.3.4 dev eth0
(for that to work, you need forwarding on, and the route for the host
must point to a different outbound device)

I agree that the proxy_arp /proc knob is pure evil. In my opinion it
should be removed in favour of "ip neighbor add proxy".

> > Why not use a VLAN?
> 
> Because I really don't need one.  Plus it doesn't matter anyway - the
> firewall can act as a router on a stick to go between my H.323 devices
> and private IP servers.  With or without VLANs makes no difference in
> this case.  

The VLAN saves you the SNAT on your clients traffic towards the NATed
services, because the traffic back from those NATed services goes
through the firewall, which will apply its conntrack entries.

Also, what you're doing is a case of _layer 3_ routing of packets that
arrive at an interface - br0 - back out to the same interface - br0. I'm
not excluding the possibility that the bridge code is broken, but my
nose tells me that your SNAT-DNAT setup is more likely to have gotten
broken.

Either way I still don't understand your setup. Are you using ebtables?
Where is your private IP that's facing towards the clients? Is there a
separate third DMZ network? What is $DMZ_IFACE?

(Posting your setup script is a noble thought completeness-wise, but
tbfh it's unreadable and maybe even the wrong parts. Please boil it
down to a few lines and please include the client direction...)

> - and now I think the problem is, bridging is supposed to turn on
> PROMISC mode and it didn't.  I had to do it by hand myself.

So it works when you switch the bridge members into PROMISC? (not the
bridge itself!)

> What I don't know yet is, is this a Fedora bug or a stock kernel bug?
> Is anyone from Red Hat following this email list?

I only know I haven't seen it anywhere, but my bridges are getting fewer
and fewer...


-David

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> First of all, I still can't find your kernel version in any of your
> mails. Can you please repeat the uname -a output of the affected box?

Woops - sorry - I never posted it, that's why you didn't see it.  Here
it is:

[root@ehac-fw2011 firewall-scripts]# uname -a
Linux ehac-fw2011 2.6.35.6-48.fc14.i686.PAE #1 SMP Fri Oct 22 15:27:53
UTC 2010 i686 i686 i386 GNU/Linux

It's just as Red Hat delivered it.


> The VLAN saves you the SNAT on your clients traffic towards the NATed
> services, because the traffic back from those NATed services goes
> through the firewall, which will apply its conntrack entries.

I don't see it that way.  I have a couple of devices with public IP
Addresses and most with "normal" private IP Addresses.  Those public IP
devices can easily be on the same Ethernet segment and in the same
collision domain as the private ones.  There's no good reason to
separate them at this particular site.  Oh - I think see what you're
thinking - the words, "public IP Address", lead to a wrong conclusion
that those devices really are **public**.  Just because some of the
devices have public IP Addresses does **NOT**  mean they're completely
accessible to the public.  Just like for, say, web servers, we NAT TCP
port 80 and block the rest - for these public IP Address devices at this
site, I ACCEPT TCP 1720 and use the H.323 conntrack module to handle
that traffic because H.323 does not get along easily with NAT.  Pretty
much the only difference between the public IP stuff and the private IP
stuff is, I NAT for the private IP stuff and just ACCEPT traffic I want
to let in/out for the public IP stuff.  There's no reason to separate
the public IP stuff and private IP stuff with VLANs.  

Anyway, at this site, the H.323 stuff is the **reason**  why I need a
bridge, so the H.323 stuff is only indirectly related to the problem.  I
have other sites with different reasons for various systems to have real
public IP Addresses. 


> Also, what you're doing is a case of _layer 3_ routing of packets that
> arrive at an interface - br0 - back out to the same interface - br0.

Yes, absolutely, when internal users need to access the NATed websites
using public IP Addresses instead of their private IP Addresses.
Classic router on a stick topology, but using DNAT and MASQUERADE.

Let me try to describe it this way.  Forget about the reason I need a
bridge.  I have a good reason this site is bridged and have now
hopefully presented a reasonable case why I need one.  

I have a public website at private IP Address 192.168.10.2.  The
firewall DNATs TCP port 80 for public IP Address aa.bb.115.151 to
192.168.10.2.  Most of this traffic will come in from the Internet.  But
I also want my **internal** users to have the same experience as the
rest of the world when viewing this website, so that traffic will come
in from the private LAN.  I have good reasons for this - one biggie is
so my internal users can see website changes as the rest of the world
sees them. I know I can simulate this with a private DNS server, but I
don't like multiple versions of DNS floating around.  So I choose to do
it with layer 3 routing and some fancy DNAT and SNAT (really
MASQUERADing) at the firewall.  

This all worked for several years, right up until I put in a replacement
based on Fedora 14.  By now, the script I have in place at this site is
battle hardened and has been in service for close to 10 years and many
hardware upgrades. 

And it broke with Fedora 14.

> Either way I still don't understand your setup. Are you using
ebtables?

Yes to ebtables.  I use ebtables to mark packets so I know which
interface they come in on.  Anything coming in from the Internet gets a
mark of 1.  Anything coming in on the LAN side gets a mark of 2.  You
have me kind of afraid to post the code....but I'll paste it in below
anyway.  btw, I posted code earlier because one reply asked me to do so.

.
.
.
echo "Flushing and zeroing all ebtables tables and chains"
$EBTABLES -t broute -F
$EBTABLES -t broute -Z
$EBTABLES -t filter -F
$EBTABLES -t filter -Z
$EBTABLES -t nat -F
$EBTABLES -t nat -Z

#
# Use ebtables to mark packets based on the in/out interface.
# 1 - (bit 0 set) for packets entering on the Internet physical
interface
# 2 - (bit 1 set) for packets entering on the trusted physical interface
# 3 - (bits 0 and 1) for packets exiting via the Internet physical
interface
# (Kernel 2.6.23 or so changed the order of iptables/ebtables going out,
so
# marking outbound packets is meaningless now.)

echo "Marking bridged packets at layer 2 for later layer 3 filtering."
$EBTABLES -t broute -A BROUTING -i $INET_IFACE \
        -j mark --mark-set 1 --mark-target CONTINUE
$EBTABLES -t broute -A BROUTING -i $TRUSTED1_IFACE \
        -j mark --mark-set 2 --mark-target CONTINUE
.
.
.

> Is there a separate third DMZ network? What is $DMZ_IFACE?

That DMZ network is not relevant and I should not have included any
reference to it.  It's eth2 and the NIC is in the box but nothing is
connected to it.  And it's not part of the bridge.  It's an empty NIC
that we may use in the future, but right now not relevant.  


> Where is your private IP that's facing towards the clients?

I don't know what this question means.  The setup is a traditional
Public<-->firewall<-->private topology, as the ASCII art I posted
earlier shows.  But some of the stuff on the private side needs public
IP Addresses, so the firewall is a bridge plus a router, not just a
router.  


> So it works when you switch the bridge members into PROMISC? (not the
> bridge itself!)

No, the br0 bridge device itself.  After a bunch of troubleshooting,
below is literally the single one and only change I needed to make this
work again.

.
.
.
echo "  Putting $BR_IFACE into promiscuous mode"
# This fixes a bug forwarding packets bound for external IP Addresses
# from the private LAN.

ip link set $BR_IFACE promisc on
.
.
.

I don't think I should need to do this by hand and I never needed it
before.  That's why it took me weeks and plenty of help with the
Netfilter folks to find it.  Something apparently changed with bridging.


Reading through some of the replies to this post, I decided to look and
see what happens with the physical ethnnn devices when I add them to a
bridge, so I looked at a couple of other sites with similar setups.
I've never had any need to dig into this before because it all just
worked.  

- Greg

Re: Bridging behavior apparently changed around the Fedora 14 time

[David Lamparter]

On Tue, Jul 12, 2011 at 09:30:05AM -0500, Greg Scott wrote:
> Linux ehac-fw2011 2.6.35.6-48.fc14.i686.PAE #1 SMP Fri Oct 22 15:27:53
> UTC 2010 i686 i686 i386 GNU/Linux
> 
> It's just as Red Hat delivered it.

Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but I
think you need to go bug RedHat.

Anyway, either someone else should have had your problem by now (so it
might be fixed) or I'd say there's something wrong with your setup
(maybe changed defaults) or RedHat messed up the kernel ;).

> > The VLAN saves you the SNAT on your clients traffic towards the NATed
> > services, because the traffic back from those NATed services goes
> > through the firewall, which will apply its conntrack entries.
> 
> I don't see it that way.  I have a couple of devices with public IP
> Addresses and most with "normal" private IP Addresses.  Those public IP
[snip]

You totally misunderstood me. I'm suggesting the separate VLAN for your
servers which have private IPs but which have services exposed to the
internet (and your clients) on public IPs through NAT.

Your H323 stuff is totally unrelated.

> > Also, what you're doing is a case of _layer 3_ routing of packets that
> > arrive at an interface - br0 - back out to the same interface - br0.
> 
> Yes, absolutely, when internal users need to access the NATed websites
> using public IP Addresses instead of their private IP Addresses.
> Classic router on a stick topology, but using DNAT and MASQUERADE.

Where's the log fire for your router on a stick? *SCNR*

> Let me try to describe it this way.  Forget about the reason I need a
> bridge.  I have a good reason this site is bridged and have now
> hopefully presented a reasonable case why I need one.  

Yes. Your problem seems to be between the private-IP clients in your
network and your private-IP servers if I understand correctly.

The bridge is most likely completely innocent and your "stick" NAT
setup just broke down due to some changed default I'd guess.

> And it broke with Fedora 14.

> > Where is your private IP that's facing towards the clients?
> 
> I don't know what this question means.  The setup is a traditional
> Public<-->firewall<-->private topology, as the ASCII art I posted
> earlier shows.  But some of the stuff on the private side needs public
> IP Addresses, so the firewall is a bridge plus a router, not just a
> router.  

Yes. And because it is a router, it as an IP from the private subnet
your clients are in. My question was: what device is that IP on?

> > So it works when you switch the bridge members into PROMISC? (not the
> > bridge itself!)
> 
> No, the br0 bridge device itself.  After a bunch of troubleshooting,
> below is literally the single one and only change I needed to make this
> work again.

This makes me think yet more that the bridge code is innocent.

> I don't think I should need to do this by hand and I never needed it
> before.  That's why it took me weeks and plenty of help with the
> Netfilter folks to find it.  Something apparently changed with bridging.

No. You're jumping to conclusions. You're affecting the "top" bridge
device's promiscuity. I would say that the effect you're seeing is in
the IP stack above it, caused by it now promiscuously handling packets
that are dropped otherwise.

Setups like yours need a lot of caution regarding ICMP redirect /
shared_media / ARP settings. Please check those.


-David


P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :)

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :)

Sorry - didn't ignore it, just didn't reply back to it.  I'll look into
it. What I've read about this before has all been kind of vague.  Does
this mean I proxy ARP only for IP Address 1.2.3.4?  So somebody sends an
ARP whois 1.2.3.4, I'll answer with 1.2.3.4. is at {My MAC Address}?  If
so, then I agree, not nearly as evil as just setting proxy_arp.  

> Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but
I
> think you need to go bug RedHat.

Yeah, maybe.  OK, probably.  This was such a bizarre problem - I started
with Netfilter and those guys suggested I try here.  At least now I
understand the problem lots better than before. And it's not like I can
just go and update dozens of kernels at dozens of sites all the time
when a new kernel comes out.  

> You totally misunderstood me. I'm suggesting the separate VLAN for
your
> servers which have private IPs but which have services exposed to the
> internet (and your clients) on public IPs through NAT.

Ahh - OK.  The challenge with many small sites is, economic reality.
That same server that hosts the public ftp and websites also hosts all
the internal Windows file/print services.  It's the only server at this
site, so it has several roles.  I would love to build a real DMZ network
and put all the public facing stuff in there, but I don't have money for
multiple servers.  This will become even more difficult to separate when
we go to virtual servers and clustered hosts.  

> Your H323 stuff is totally unrelated.

Agreed.  Wholeheartedly.  

> Yes. Your problem seems to be between the private-IP clients in your
> network and your private-IP servers if I understand correctly.

Yes.  Dead-bang, right on target.  

> Yes. And because it is a router, it as an IP from the private subnet
> your clients are in. My question was: what device is that IP on?

Ahh - eth1 is the private LAN side, 192.168.10.1.  All the NATed LAN
stuff and all the workstations are in the 192.168.10.0/24 subnet and
connected to eth1.  Eth0 is the Internet side.  The Internet side has
the firewall NIC, a cable, and the Internet router.  That's it.
Everything is connected to the LAN side.  

> No. You're jumping to conclusions. You're affecting the "top" bridge
> device's promiscuity. I would say that the effect you're seeing is in
> the IP stack above it, caused by it now promiscuously handling packets
> that are dropped otherwise.

Well they were sure dropped before I set it to PROMISC mode, that's for
sure. And it all worked with the earlier version.  That's why this feels
like a layer 2 issue.  If it was an IP issue, why didn't it break
several years ago when I first set it up?

Does bridging make everything a little more complex and delicate to set
up?  Well, yeah.  And some of the netfilter stuff has been a moving
target over the years.  

I don't see how ICMP redirects matter.  Comparing
/proc/sys/net/ipv4/conf/*/accept_redirects with this version and an
older one at another site - all identical.  ../all/accept_recdirects is
0, the rest are all 1.  Shared media and ARP settings -
/proc/sys/net/ipv4/conf/*/shared_media - all 1 for all interfaces.
There are a zillion arp settings.  Looking at
/proc/sys/net/ipv4/conf/*/*arp* - all are 0 in both the other older site
and this newer site.  

Curiously - at one of my other older sites, apparently br0 is not in
promisc mode.  But I don't think these guys do any of the stick routing
stuff.  I wonder if these guys have the problem but we don't see it
because they never try it?

[root@NSSSS-fw1 ~]# more /sys/class/net/br0/flags
0x1003
[root@NSSSS-fw1 ~]#
[root@NSSSS-fw1 ~]# more /proc/version
Linux version 2.6.32.11-99.fc12.i686.PAE
(mockbuild@x86-05.phx2.fedoraproject.org) (gcc version 4.4.3 20100127
(Red Hat 4.4.3-4) (GCC) )
#1 SMP Mon Apr 5 16:15:03 EDT 2010
[root@NSSSS-fw1 ~]#
[root@NSSSS-fw1 ~]# uname -a
Linux NSSSS-fw1 2.6.32.11-99.fc12.i686.PAE #1 SMP Mon Apr 5 16:15:03 EDT
2010 i686 i686 i386 GNU/Linux
[root@NSSSS-fw1 ~]#


Here is a much older bridged site based on Fedora 9 and I'm sure these
guys use my stick routing stuff.  Look at the difference in ..br0/flags.


[root@lme-fw2 ~]#  more /sys/class/net/br0/flags
0x1103
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# more /proc/version
Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0
20080428 (Red H
at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# uname -a
Linux lme-fw2 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686
i686 i386 GNU/Linux

I can still get my hands on the old box at the site in question.  I
guess it couldn't hurt to fire it up and look at its br0 flags.  

- Greg

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

Aw nuts, nothing is ever straightforward.  

When I do:

ip link set br0 promisc on

My internal users can see the internally hosted websites using the
public IP Addresses.  The router on a stick rules I put in work just
fine.  (In on br0/eth1, DNATed in PREROUTING, MASQUERADEd in
POSTROUTING, back out br0/eth1 to the correct internal host.)

However, I just learned tonight, this breaks both inbound and outbound
PPTP VPNs.  And when I do:

ip link set br0 promisc off

now my PPTP VPNs work, but this breaks my above router on a stick rules.


My PPTP VPN stuff uses the GRE iptables conntrack modules,
ip_conntrack_pptp and ip_nat_pptp, and some PREROUTING and POSTROUTING
rules to DNAT TCP 1723 and all GRE packets to an internal Windows RRAS
server.  But when I turn promisc on for br0, I see a storm of packets
looping over and over again, until the remote client finally times out
after what seems like an eternity.  

I'll bet that bridge forwards my packets out the wrong physical ethnn
interface when it's in promisc mode and that's why my NATed PPTP VPN
breaks.  Which makes me wonder if putting br0 in promisc mode breaks any
of my other NATed services.

- Greg Scott



-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Greg Scott
Sent: Tuesday, July 12, 2011 11:29 AM
To: David Lamparter
Cc: netdev@vger.kernel.org; Lynn Hanson; Joe Whalen
Subject: RE: Bridging behavior apparently changed around the Fedora 14
time


> P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :)

Sorry - didn't ignore it, just didn't reply back to it.  I'll look into
it. What I've read about this before has all been kind of vague.  Does
this mean I proxy ARP only for IP Address 1.2.3.4?  So somebody sends an
ARP whois 1.2.3.4, I'll answer with 1.2.3.4. is at {My MAC Address}?  If
so, then I agree, not nearly as evil as just setting proxy_arp.  

> Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but
I
> think you need to go bug RedHat.

Yeah, maybe.  OK, probably.  This was such a bizarre problem - I started
with Netfilter and those guys suggested I try here.  At least now I
understand the problem lots better than before. And it's not like I can
just go and update dozens of kernels at dozens of sites all the time
when a new kernel comes out.  

> You totally misunderstood me. I'm suggesting the separate VLAN for
your
> servers which have private IPs but which have services exposed to the
> internet (and your clients) on public IPs through NAT.

Ahh - OK.  The challenge with many small sites is, economic reality.
That same server that hosts the public ftp and websites also hosts all
the internal Windows file/print services.  It's the only server at this
site, so it has several roles.  I would love to build a real DMZ network
and put all the public facing stuff in there, but I don't have money for
multiple servers.  This will become even more difficult to separate when
we go to virtual servers and clustered hosts.  

> Your H323 stuff is totally unrelated.

Agreed.  Wholeheartedly.  

> Yes. Your problem seems to be between the private-IP clients in your
> network and your private-IP servers if I understand correctly.

Yes.  Dead-bang, right on target.  

> Yes. And because it is a router, it as an IP from the private subnet
> your clients are in. My question was: what device is that IP on?

Ahh - eth1 is the private LAN side, 192.168.10.1.  All the NATed LAN
stuff and all the workstations are in the 192.168.10.0/24 subnet and
connected to eth1.  Eth0 is the Internet side.  The Internet side has
the firewall NIC, a cable, and the Internet router.  That's it.
Everything is connected to the LAN side.  

> No. You're jumping to conclusions. You're affecting the "top" bridge
> device's promiscuity. I would say that the effect you're seeing is in
> the IP stack above it, caused by it now promiscuously handling packets
> that are dropped otherwise.

Well they were sure dropped before I set it to PROMISC mode, that's for
sure. And it all worked with the earlier version.  That's why this feels
like a layer 2 issue.  If it was an IP issue, why didn't it break
several years ago when I first set it up?

Does bridging make everything a little more complex and delicate to set
up?  Well, yeah.  And some of the netfilter stuff has been a moving
target over the years.  

I don't see how ICMP redirects matter.  Comparing
/proc/sys/net/ipv4/conf/*/accept_redirects with this version and an
older one at another site - all identical.  ../all/accept_recdirects is
0, the rest are all 1.  Shared media and ARP settings -
/proc/sys/net/ipv4/conf/*/shared_media - all 1 for all interfaces.
There are a zillion arp settings.  Looking at
/proc/sys/net/ipv4/conf/*/*arp* - all are 0 in both the other older site
and this newer site.  

Curiously - at one of my other older sites, apparently br0 is not in
promisc mode.  But I don't think these guys do any of the stick routing
stuff.  I wonder if these guys have the problem but we don't see it
because they never try it?

[root@NSSSS-fw1 ~]# more /sys/class/net/br0/flags
0x1003
[root@NSSSS-fw1 ~]#
[root@NSSSS-fw1 ~]# more /proc/version
Linux version 2.6.32.11-99.fc12.i686.PAE
(mockbuild@x86-05.phx2.fedoraproject.org) (gcc version 4.4.3 20100127
(Red Hat 4.4.3-4) (GCC) )
#1 SMP Mon Apr 5 16:15:03 EDT 2010
[root@NSSSS-fw1 ~]#
[root@NSSSS-fw1 ~]# uname -a
Linux NSSSS-fw1 2.6.32.11-99.fc12.i686.PAE #1 SMP Mon Apr 5 16:15:03 EDT
2010 i686 i686 i386 GNU/Linux
[root@NSSSS-fw1 ~]#


Here is a much older bridged site based on Fedora 9 and I'm sure these
guys use my stick routing stuff.  Look at the difference in ..br0/flags.


[root@lme-fw2 ~]#  more /sys/class/net/br0/flags
0x1103
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# more /proc/version
Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0
20080428 (Red H
at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# uname -a
Linux lme-fw2 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686
i686 i386 GNU/Linux

I can still get my hands on the old box at the site in question.  I
guess it couldn't hurt to fire it up and look at its br0 flags.  

- Greg

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

Aw nuts, nothing is ever straightforward.  

When I do:

ip link set br0 promisc on

My internal users can see the internally hosted websites using the
public IP Addresses.  The router on a stick rules I put in work just
fine.  (In on br0/eth1, DNATed in PREROUTING, MASQUERADEd in
POSTROUTING, back out br0/eth1 to the correct internal host.)

However, I just learned last night, this breaks both inbound and
outbound PPTP VPNs.  And when I do:

ip link set br0 promisc off

now my PPTP VPNs work, but this breaks my above router on a stick rules.


My PPTP VPN stuff uses the GRE iptables conntrack modules,
ip_conntrack_pptp and ip_nat_pptp, and some PREROUTING and POSTROUTING
rules to DNAT TCP 1723 and all GRE packets to an internal Windows RRAS
server.  But when I turn promisc on for br0, I see a storm of packets
looping over and over again, until the remote client finally times out
after what seems like an eternity.  

I'll bet that bridge forwards my packets out the wrong physical ethnn
interface when it's in promisc mode and that's why my NATed PPTP VPN
breaks.  Which makes me wonder if putting br0 in promisc mode breaks any
of my other NATed services.

So back to square one I guess.

- Greg Scott



-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Greg Scott
Sent: Tuesday, July 12, 2011 11:29 AM
To: David Lamparter
Cc: netdev@vger.kernel.org; Lynn Hanson; Joe Whalen
Subject: RE: Bridging behavior apparently changed around the Fedora 14
time


> P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :)

Sorry - didn't ignore it, just didn't reply back to it.  I'll look into
it. What I've read about this before has all been kind of vague.  Does
this mean I proxy ARP only for IP Address 1.2.3.4?  So somebody sends an
ARP whois 1.2.3.4, I'll answer with 1.2.3.4. is at {My MAC Address}?  If
so, then I agree, not nearly as evil as just setting proxy_arp.  

> Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but
I
> think you need to go bug RedHat.

Yeah, maybe.  OK, probably.  This was such a bizarre problem - I started
with Netfilter and those guys suggested I try here.  At least now I
understand the problem lots better than before. And it's not like I can
just go and update dozens of kernels at dozens of sites all the time
when a new kernel comes out.  

> You totally misunderstood me. I'm suggesting the separate VLAN for
your
> servers which have private IPs but which have services exposed to the
> internet (and your clients) on public IPs through NAT.

Ahh - OK.  The challenge with many small sites is, economic reality.
That same server that hosts the public ftp and websites also hosts all
the internal Windows file/print services.  It's the only server at this
site, so it has several roles.  I would love to build a real DMZ network
and put all the public facing stuff in there, but I don't have money for
multiple servers.  This will become even more difficult to separate when
we go to virtual servers and clustered hosts.  

> Your H323 stuff is totally unrelated.

Agreed.  Wholeheartedly.  

> Yes. Your problem seems to be between the private-IP clients in your
> network and your private-IP servers if I understand correctly.

Yes.  Dead-bang, right on target.  

> Yes. And because it is a router, it as an IP from the private subnet
> your clients are in. My question was: what device is that IP on?

Ahh - eth1 is the private LAN side, 192.168.10.1.  All the NATed LAN
stuff and all the workstations are in the 192.168.10.0/24 subnet and
connected to eth1.  Eth0 is the Internet side.  The Internet side has
the firewall NIC, a cable, and the Internet router.  That's it.
Everything is connected to the LAN side.  

> No. You're jumping to conclusions. You're affecting the "top" bridge
> device's promiscuity. I would say that the effect you're seeing is in
> the IP stack above it, caused by it now promiscuously handling packets
> that are dropped otherwise.

Well they were sure dropped before I set it to PROMISC mode, that's for
sure. And it all worked with the earlier version.  That's why this feels
like a layer 2 issue.  If it was an IP issue, why didn't it break
several years ago when I first set it up?

Does bridging make everything a little more complex and delicate to set
up?  Well, yeah.  And some of the netfilter stuff has been a moving
target over the years.  

I don't see how ICMP redirects matter.  Comparing
/proc/sys/net/ipv4/conf/*/accept_redirects with this version and an
older one at another site - all identical.  ../all/accept_recdirects is
0, the rest are all 1.  Shared media and ARP settings -
/proc/sys/net/ipv4/conf/*/shared_media - all 1 for all interfaces.
There are a zillion arp settings.  Looking at
/proc/sys/net/ipv4/conf/*/*arp* - all are 0 in both the other older site
and this newer site.  

Curiously - at one of my other older sites, apparently br0 is not in
promisc mode.  But I don't think these guys do any of the stick routing
stuff.  I wonder if these guys have the problem but we don't see it
because they never try it?

[root@NSSSS-fw1 ~]# more /sys/class/net/br0/flags
0x1003
[root@NSSSS-fw1 ~]#
[root@NSSSS-fw1 ~]# more /proc/version
Linux version 2.6.32.11-99.fc12.i686.PAE
(mockbuild@x86-05.phx2.fedoraproject.org) (gcc version 4.4.3 20100127
(Red Hat 4.4.3-4) (GCC) )
#1 SMP Mon Apr 5 16:15:03 EDT 2010
[root@NSSSS-fw1 ~]#
[root@NSSSS-fw1 ~]# uname -a
Linux NSSSS-fw1 2.6.32.11-99.fc12.i686.PAE #1 SMP Mon Apr 5 16:15:03 EDT
2010 i686 i686 i386 GNU/Linux
[root@NSSSS-fw1 ~]#


Here is a much older bridged site based on Fedora 9 and I'm sure these
guys use my stick routing stuff.  Look at the difference in ..br0/flags.


[root@lme-fw2 ~]#  more /sys/class/net/br0/flags
0x1103
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# more /proc/version
Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0
20080428 (Red H
at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008
[root@lme-fw2 ~]#
[root@lme-fw2 ~]# uname -a
Linux lme-fw2 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686
i686 i386 GNU/Linux

I can still get my hands on the old box at the site in question.  I
guess it couldn't hurt to fire it up and look at its br0 flags.  

- Greg

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

> Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but
I
> think you need to go bug RedHat.

Per your suggestion, I just put together a Red Hat bugzilla report and
documented the behavior to the best of my ability.  It's listed as bug
#724862 in bugzilla.redhat.com.

- Greg

RE: Bridging behavior apparently changed around the Fedora 14 time

[Greg Scott]

Hang on a second - David, I think you're trying to tell me I'm the
problem.  Maybe so...

I send a frame in on eth1/br0.  It traverses a bunch of ebtables and
iptables DNAT and SNAT rules and then wants to come back out on
eth1/br0.  But this is both a bridge and a router, and trying to
remember my bridge basics - as I recall, bridges might not want to
forward frames back out the same physical interface they came in on.  

After running this frame/packet through a gauntlet of ebtables/iptables
rules and NATing the snot out of the IP packet inside the frame, I
wonder if it's still the same layer 2 frame when it gets ready to come
back out on eth1?  Even though NAT changed the layer 3 routed packet IP
info, I wonder if it's still the same layer 2 datalink frame?  So the
bridge says, "No way Jose, that's the same physical interface you came
in on" and drops it.  

And then turning on promisc mode makes the bridge less picky and it
happily forwards all frames everywhere.  

Is this what you've been trying to tell me and I'm too thickheaded to
get it?

If so, to work around the problem, I was reading about some ebtables
broute ACCEPT/DROP targets that are apparently named badly - DROP
apparently tells the bridge to route the packet instead of bridging the
frame, while ACCEPT tells the bridge to bridge it.  Should I be looking
in this direction?

But thinking about this some more - there are cases, such as cascaded
switches, or hubs behind a switch, where you want a bridge to forward
frames back out the same interface.  Well . . . maybe.  There's still
something about forwarding out the same interface a frame comes in on
that's bugging me.

- Greg

Very confused about broute DROP

[Greg Scott]

I don't get this.  Why does:

ebtables -t broute -A BROUTING -j DROP

completely knock a Linux host offline?

This is what the man page for ebtables says:

The targets DROP and ACCEPT have a special meaning in the broute table
(these names are used instead of more descriptive  names  to  keep the
implementation  generic).   DROP  actually means the frame has to be
routed, while ACCEPT means the frame has to be bridged. The BROUTING
chain is traversed very early. However, it is  only  traversed  by
frames  entering  on  a bridge port that is in forwarding state.
Normally those frames would be bridged, but you can decide otherwise
here. The redirect target is very handy here.

So based on the above paragraph, I should be able to do something like
this:

# Here is what to bridge
ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP1 -j
ACCEPT
ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP2 -j
ACCEPT

# Route everything else
ebtables -t broute -A BROUTING -j DROP

So I tried above and knocked that box completely offline.  I'm missing
something.

Here is what the paragraph about redirect in the ebtables man pages
says: 

The  redirect target will change the MAC target address to that of the
bridge device the frame arrived on. This target can only be used in the
BROUTING chain of the broute table and the PREROUTING chain of  the  nat
table.  In  the  BROUTING  chain,  the MAC address of the bridge port is
used as destination address, in the PREROUTING chain, the MAC address of
the bridge is used.

OK - so this target MAC address - is this the MAC Address of an ethnn
port that's part of the bridge, or the MAC Address of another node?  I
was thinking it was the MAC Address of another node, but maybe it's just
the MAC Address of a port on this bridge?  

And there are some examples here:
http://ebtables.sourceforge.net/examples/basic.html#ex_redirect

that I really don't get.  So instead of trial and error guessing, I
figured I would ask.  

If anyone can help me understand this, I'll take a stab at writing it up
as clearly as I know how for use in future versions of man pages.  

Thanks

- Greg Scott

RE: Very confused about broute DROP

[Christian Benvenuti (benve)]

How about ARP? You need it too ...

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Greg Scott
> Sent: Thursday, September 15, 2011 3:48 PM
> To: netdev@vger.kernel.org
> Cc: Graham Parenteau
> Subject: Very confused about broute DROP
> 
> I don't get this.  Why does:
> 
> ebtables -t broute -A BROUTING -j DROP
> 
> completely knock a Linux host offline?
> 
> This is what the man page for ebtables says:
> 
> The targets DROP and ACCEPT have a special meaning in the broute table
> (these names are used instead of more descriptive  names  to  keep the
> implementation  generic).   DROP  actually means the frame has to be
> routed, while ACCEPT means the frame has to be bridged. The BROUTING
> chain is traversed very early. However, it is  only  traversed  by
> frames  entering  on  a bridge port that is in forwarding state.
> Normally those frames would be bridged, but you can decide otherwise
> here. The redirect target is very handy here.
> 
> So based on the above paragraph, I should be able to do something like
> this:
> 
> # Here is what to bridge
> ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP1 -j
> ACCEPT
> ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP2 -j
> ACCEPT
> 
> # Route everything else
> ebtables -t broute -A BROUTING -j DROP
> 
> So I tried above and knocked that box completely offline.  I'm missing
> something.
> 
> Here is what the paragraph about redirect in the ebtables man pages
> says:
> 
> The  redirect target will change the MAC target address to that of the
> bridge device the frame arrived on. This target can only be used in
the
> BROUTING chain of the broute table and the PREROUTING chain of  the
> nat
> table.  In  the  BROUTING  chain,  the MAC address of the bridge port
> is
> used as destination address, in the PREROUTING chain, the MAC address
> of
> the bridge is used.
> 
> OK - so this target MAC address - is this the MAC Address of an ethnn
> port that's part of the bridge, or the MAC Address of another node?  I
> was thinking it was the MAC Address of another node, but maybe it's
> just
> the MAC Address of a port on this bridge?
> 
> And there are some examples here:
> http://ebtables.sourceforge.net/examples/basic.html#ex_redirect
> 
> that I really don't get.  So instead of trial and error guessing, I
> figured I would ask.
> 
> If anyone can help me understand this, I'll take a stab at writing it
> up
> as clearly as I know how for use in future versions of man pages.
> 
> Thanks
> 
> - Greg Scott
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Very confused about broute DROP

[Greg Scott]

If I'm reading the ebtables ARP stuff right, it looks like I can use
this for selective proxy ARPs.  This may have made my life much nicer
with a project that didn't work out very well 5 years ago when I
inherited a network with systems that had public IP Addresses on both
sides of the firewall. For now, why did 

ebtables -t broute -A BROUTING -j DROP

shut my system down and force me to drive over there at 4AM this morning
to undo it?  It's not supposed to really drop, it's supposed to route
it.  And those examples aren't making any sense to me.  

Thanks

- Greg



-----Original Message-----
From: Christian Benvenuti (benve) [mailto:benve@cisco.com] 
Sent: Thursday, September 15, 2011 6:08 PM
To: Greg Scott; netdev@vger.kernel.org
Cc: Graham Parenteau
Subject: RE: Very confused about broute DROP

How about ARP? You need it too ...

RE: Very confused about broute DROP

[Christian Benvenuti (benve)]

What I meant is that your host needs to be able to route
(which means ... to process) its own ARP traffic, ... IPv4
does not work without ARP, right?
This means you need to add one more DROP rule for the ARP
traffic that is addressed to the MAC of the host interfaces
(nothing to do with proxy ARP).

/Chris

> -----Original Message-----
> From: Greg Scott [mailto:GregScott@Infrasupport.com]
> Sent: Thursday, September 15, 2011 8:19 PM
> To: Christian Benvenuti (benve); netdev@vger.kernel.org
> Cc: Graham Parenteau
> Subject: RE: Very confused about broute DROP
> 
> If I'm reading the ebtables ARP stuff right, it looks like I can use
> this for selective proxy ARPs.  This may have made my life much nicer
> with a project that didn't work out very well 5 years ago when I
> inherited a network with systems that had public IP Addresses on both
> sides of the firewall. For now, why did
> 
> ebtables -t broute -A BROUTING -j DROP
> 
> shut my system down and force me to drive over there at 4AM this
> morning
> to undo it?  It's not supposed to really drop, it's supposed to route
> it.  And those examples aren't making any sense to me.
> 
> Thanks
> 
> - Greg
> 
> 
> 
> -----Original Message-----
> From: Christian Benvenuti (benve) [mailto:benve@cisco.com]
> Sent: Thursday, September 15, 2011 6:08 PM
> To: Greg Scott; netdev@vger.kernel.org
> Cc: Graham Parenteau
> Subject: RE: Very confused about broute DROP
> 
> How about ARP? You need it too ...
> 

RE: Very confused about broute DROP

[Greg Scott]

Definitely counter-intuitive in my head.  Three little ebtables rules -
ACCEPT anything IPv4 - which includes ARPs - for a couple of IP
Addresses and then DROP everything else for all protocols.  (Which
really means bridge for those IPv4 Addresses and route for everything
else.)

I don't see how a rule specific to ARPs matters here.  ARP is an IPv4
protocol and should be implied in the rules I set up - right?  Why a
rule specific to ARPs when it's already part of IPv4?

This just hit me - layer 2 is stateless.  I have ACCEPT rules for frames
bound for **destination** IPv4 IP Addresses.  I wonder if I need similar
rules for these same IP Addresses as sources?  But that still doesn't
make sense - that rule to DROP everything else covers all sources and
all destinations.  When I put in that ebtables DROP rule, the box turns
into a black hole instead of a router.  

Maybe one day the lightbulb will light up in my head, but right now I
still don't get it.

- Greg



-----Original Message-----
From: Christian Benvenuti (benve) [mailto:benve@cisco.com] 
Sent: Thursday, September 15, 2011 11:23 PM
To: Greg Scott; netdev@vger.kernel.org
Cc: Graham Parenteau
Subject: RE: Very confused about broute DROP

What I meant is that your host needs to be able to route
(which means ... to process) its own ARP traffic, ... IPv4
does not work without ARP, right?
This means you need to add one more DROP rule for the ARP
traffic that is addressed to the MAC of the host interfaces
(nothing to do with proxy ARP).

/Chris

RE: Very confused about broute DROP

[Greg Scott]

A clarification - hopefully my embarrassing mistake can be of some use.
ARP is an L2 protocol, not an L3 protocol.  ARP is  **not** part of IPv4
or IPv6.  Look in the IANA list of IP protocol numbers - there's none
for ARP.  ARP is defined in Layer 2, not L3.  I would have missed that
question on a cert test.  That's why my broute rules need to take ARPs
into account.  

- Greg