* Bridging behavior apparently changed around the Fedora 14 time @ 2011-07-11 18:25 Greg Scott 2011-07-11 20:07 ` Stephen Hemminger 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-11 18:25 UTC (permalink / raw) To: netdev; +Cc: Lynn Hanson, Joe Whalen I ran into a strange situation - I am using a firewall set up as a bridge. Physical device eth1 is the private LAN side, eth0 is the public Internet side. I setup bridge br0 to bridge eth0 and eth1 together. I need a bridge because this site has a couple of nodes on the LAN side that need real public IP Addresses. This site also has a few web and ftp sites. These are NATed behind the firewall, but internal users need to see them the same way as the rest of the world. So I use some iptables SNAT and DNAT rules to make this happen. Device br0 has the relevant public IP Address(es) and then NATs to the appropriate private IP Address(es). The ruleset works and the system has been up and running for several years. I recently replaced the old system with a new one running Fedora 14 and that's when the weird behavior started. Now, when internal people try to look at those web/ftp sites using the public IP Addresses, they get nowhere. Unless I watch with tcpdump - and then while I'm watching , all works as it should. With some help, we figured out the reason it works when watching with tcpdump - because tcpdump puts the device being monitored into promiscuous mode. And, sure enough, when I do: ip link set br0 promisc on everything works as it should. Looking at "ip link show", it looks like bridge br0 takes on the MAC address of physical NIC eth0. But the internal LAN is connected to physical eth1. I wonder if this behavior is different than the older version? If the MAC Address for bridge br0 is different than the physical device I'm actually connected to, I wonder if bridging "thinks" I'm trying to hit a foreign MAC Address - especially since I'm doing both SNAT and DNAT on the same packet? [root@ehac-fw2011 ~]# ip link show eth1 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0d:88:31:d8:24 brd ff:ff:ff:ff:ff:ff [root@ehac-fw2011 ~]# ip link show eth0 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff [root@ehac-fw2011 ~]# [root@ehac-fw2011 ~]# ip link show br0 5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc prio state UNKNOWN link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff [root@ehac-fw2011 ~]# brctl show macs br0 bridge name bridge id STP enabled interfaces br0 8000.0003473a5979 no eth0 eth1 Hmmmm - so a packet comes in on eth1, with a destination MAC Address belonging to physical eth0. So eth1 throws it away because it "thinks" this is a foreign MAC Address? But this all worked before, so what's different? Or were earlier bridges in promiscuous mode by default and now they're not? Have I stumbled across a new bridging bug? Is the best workaround to just put br0 into promiscuous mode? Thanks - Greg Scott ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 18:25 Bridging behavior apparently changed around the Fedora 14 time Greg Scott @ 2011-07-11 20:07 ` Stephen Hemminger 2011-07-11 20:41 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Stephen Hemminger @ 2011-07-11 20:07 UTC (permalink / raw) To: Greg Scott; +Cc: netdev, Lynn Hanson, Joe Whalen On Mon, 11 Jul 2011 13:25:46 -0500 "Greg Scott" <GregScott@Infrasupport.com> wrote: > I ran into a strange situation - I am using a firewall set up as a > bridge. Physical device eth1 is the private LAN side, eth0 is the > public Internet side. I setup bridge br0 to bridge eth0 and eth1 > together. I need a bridge because this site has a couple of nodes on > the LAN side that need real public IP Addresses. > > This site also has a few web and ftp sites. These are NATed behind the > firewall, but internal users need to see them the same way as the rest > of the world. So I use some iptables SNAT and DNAT rules to make this > happen. Device br0 has the relevant public IP Address(es) and then NATs > to the appropriate private IP Address(es). The ruleset works and the > system has been up and running for several years. > > I recently replaced the old system with a new one running Fedora 14 and > that's when the weird behavior started. > > Now, when internal people try to look at those web/ftp sites using the > public IP Addresses, they get nowhere. Unless I watch with tcpdump - > and then while I'm watching , all works as it should. With some help, > we figured out the reason it works when watching with tcpdump - because > tcpdump puts the device being monitored into promiscuous mode. > > And, sure enough, when I do: > ip link set br0 promisc on > > everything works as it should. Please provide more configuration information like: - NIC type - iptables and bridge and address configuration Do you have reverse path filtering enabled/disabled? ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 20:07 ` Stephen Hemminger @ 2011-07-11 20:41 ` Greg Scott 2011-07-11 20:49 ` Stephen Hemminger 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-11 20:41 UTC (permalink / raw) To: Stephen Hemminger; +Cc: netdev, Lynn Hanson, Joe Whalen > Please provide more configuration information like: > - NIC type > - iptables and bridge and address configuration The NICs are 3Com 3C905B's. The iptables config is complex - I'll paste in the relevant pieces. Note that it all works as expected when I put the br0 device into promiscuous mode. I included all the layer 2 MAC address info in the original email. > Do you have reverse path filtering enabled/disabled? rp_filter doesn't seem to effect the behavior one way or the other. I've tried with rp_filter set to both 0 and 1 on eth0, eth1, and br0 with no change in behavior. Here are the relevant parts of my rc.firewall script: . . . echo "Setting up aliases for public IP addresses" /sbin/ifconfig ${BR_IFACE}:0 $PUBLIC_GRE_IP netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS #/sbin/ifconfig ${BR_IFACE}:1 $PUBLIC_EMAIL_IP netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS # EMAIL and GRE share the same IP Address /sbin/ifconfig ${BR_IFACE}:2 ${PUBLIC_FTP_IP[0]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS /sbin/ifconfig ${BR_IFACE}:3 ${PUBLIC_FTP_IP[1]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS /sbin/ifconfig ${BR_IFACE}:4 ${PUBLIC_FTP_IP[2]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS /sbin/ifconfig ${BR_IFACE}:5 ${PUBLIC_FTP_IP[3]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS #/sbin/ifconfig ${BR_IFACE}:6 ${PUBLIC_VTC_IP[0]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS #/sbin/ifconfig ${BR_IFACE}:7 ${PUBLIC_VTC_IP[1]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS # The VTC devices are now bridged. They have their own IP Addresses. #/sbin/ifconfig ${BR_IFACE}:8 ${PUBLIC_WEB_IP[0]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS #/sbin/ifconfig ${BR_IFACE}:9 ${PUBLIC_WEB_IP[1]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS #/sbin/ifconfig ${BR_IFACE}:10 ${PUBLIC_WEB_IP[2]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS #/sbin/ifconfig ${BR_IFACE}:11 ${PUBLIC_WEB_IP[3]} netmask $INET_NETMASK broadcast $INET_BCAST_ADDRESS # Web and FTP sites have the same addresses. echo "ARPinging the world so everyone knows our MAC Addresses" /sbin/arping -A $INET_IP -c 1 -I $BR_IFACE /sbin/arping -A $PUBLIC_GRE_IP -c 1 -I $BR_IFACE for ADR in "${PUBLIC_FTP_IP[@]}"; do /sbin/arping -A $ADR -c 1 -I $BR_IFACE done for ADR in "${PUBLIC_WWW_IP[@]}"; do /sbin/arping -A $ADR -c 1 -I $BR_IFACE done /sbin/arping -A $TRUSTED1_IP -c 1 -I $BR_IFACE /sbin/arping -A $DMZ_IP -c 1 -I $DMZ_IFACE . . . echo " MASQUERADING requests to the internally hosted server apps" echo " (Email - SMTP, POP3, IMAP, OWA)" $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 25 -d $PRIVATE_EMAIL_IP -j MASQUERADE $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 110 -d $PRIVATE_EMAIL_IP -j MASQUERADE $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 143 -d $PRIVATE_EMAIL_IP -j MASQUERADE $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 80 -d $PRIVATE_EMAIL_IP -j MASQUERADE $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 443 -d $PRIVATE_EMAIL_IP -j MASQUERADE echo " (Internally hosted ftp sites)" for ADR in "${PUBLIC_FTP_IP[@]}"; do $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 21 -d $ADR -j MASQUERADE done echo " (Internally hosted web sites)" for ADR in "${PUBLIC_WWW_IP[@]}"; do $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 80 -d $ADR -j MASQUERADE $IPTABLES -t nat -A POSTROUTING -s $TRUSTED1_IP_RANGE -o $BR_IFACE \ -p TCP --dport 443 -d $ADR -j MASQUERADE done . . . echo " FTP sites (port 21 inbound)" for ADR in "${PRIVATE_FTP_IP[@]}"; do $IPTABLES -A FORWARD -p TCP --dport 21 -s 0/0 -d $ADR -j allowed done echo " Web (HTTP) sites (port 80 inbound)" for ADR in "${PRIVATE_WWW_IP[@]}"; do $IPTABLES -A FORWARD -p TCP --dport 80 -s 0/0 -d $ADR -j allowed done . . . echo " FTP sites (TCP port 21)" for ((i = 0 ; i < ${#PUBLIC_FTP_IP[@]} ; i++ )) do $IPTABLES -t nat -A PREROUTING -d ${PUBLIC_FTP_IP[i]} \ -p tcp --dport 21 -j DNAT --to ${PRIVATE_FTP_IP[i]} # -m mark --mark 1 (from the Internet) removed so we # DNAT for internal users also. done echo " Web (HTTP) sites (TCP port 80)" for ((i = 0 ; i < ${#PUBLIC_WWW_IP[@]} ; i++ )) do $IPTABLES -t nat -A PREROUTING -d ${PUBLIC_WWW_IP[i]} \ -p tcp --dport 80 -j DNAT --to ${PRIVATE_WWW_IP[i]} # -m mark --mark 1 (from the Internet) removed so we # DNAT for internal users also. done echo " Email - SMTP, POP3, IMAP, OWA; provision for internal users" $IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \ -p tcp --dport 25 -j DNAT --to $PRIVATE_EMAIL_IP $IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \ -p tcp --dport 110 -j DNAT --to $PRIVATE_EMAIL_IP $IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \ -p tcp --dport 143 -j DNAT --to $PRIVATE_EMAIL_IP $IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \ -p tcp --dport 80 -j DNAT --to $PRIVATE_EMAIL_IP $IPTABLES -t nat -A PREROUTING -d $PUBLIC_EMAIL_IP \ -p tcp --dport 443 -j DNAT --to $PRIVATE_EMAIL_IP As a benchmark test, from an internal host, I'll do telnet {public IP} 80. And then listen in another window on the firewall with tcpdump. Once we came across the promiscuous mode theory, we tried tcpdump with -e and -p. -e shows layer 2 stuff and -p does not put the device into promiscuous mode. Here is what we found: For my testing, from a CMD window on the server hosting the website, I do: telnet aa.bb.115.151 80 (aa.bb obfuscated first 2 IP Address octets), and then watch with tcpdump on the firewall. Also on the firewall, I did this: echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter echo "0" > /proc/sys/net/ipv4/conf/br0/rp_filter So that turned of any possible rp_filtering. And then on the firewall: /usr/sbin/tcpdump -p -e -i br0 host aa.bb.115.151 -nn -vv while the telnet was running in another window. Nothing - no output, no matter what value I use for any of the rp_filter files. Nothing from my telnet session to port 80 hits bridge br0 on the firewall. But here's the curious part - looking at physical interface eth1, I see these packets when I do the same telnet test: [root@ehac-fw2011 ~]# /usr/sbin/tcpdump -p -e -i eth1 host aa.bb.115.151 -nn -vv tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 14:51:33.412280 00:0f:20:f7:06:18 > 00:03:47:3a:59:79, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 18631, offset 0, flags [DF], proto TCP (6), length 48) 192.168.10.2.54092 > aa.bb.115.151.80: Flags [S], cksum 0xddb2 (correct), seq 4146878900, win 65535, options [mss 1460,nop,nop,sackOK], length 0 14:51:39.427928 00:0f:20:f7:06:18 > 00:03:47:3a:59:79, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 128, id 18733, offset 0, flags [DF], proto TCP (6), length 48) 192.168.10.2.54092 > aa.bb.115.151.80: Flags [S], cksum 0xddb2 (correct), seq 4146878900, win 65535, options [mss 1460,nop,nop,sackOK], length 0 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [root@ehac-fw2011 ~]# So for some reason, the bridging stuff is apparently not passing from eth1 to br0. The single, one and only change to make this work is: ip link set br0 promisc on When I do above, then the internal packets to external IP Addresses flow out to the firewall and back in as expected. Did bridging used to put the bridge device into promiscuous mode by default and now it doesn't? Just trying to figure out what's different. thanks - Greg Scott ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 20:41 ` Greg Scott @ 2011-07-11 20:49 ` Stephen Hemminger 2011-07-11 21:08 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Stephen Hemminger @ 2011-07-11 20:49 UTC (permalink / raw) To: Greg Scott; +Cc: netdev, Lynn Hanson, Joe Whalen On Mon, 11 Jul 2011 15:41:35 -0500 "Greg Scott" <GregScott@Infrasupport.com> wrote: > The NICs are 3Com 3C905B's. The iptables config is complex - I'll paste > in the relevant pieces. Note that it all works as expected when I put > the br0 device into promiscuous mode. I included all the layer 2 MAC > address info in the original email. > > > Do you have reverse path filtering enabled/disabled? > > rp_filter doesn't seem to effect the behavior one way or the other. > I've tried with rp_filter set to both 0 and 1 on eth0, eth1, and br0 > with no change in behavior. What about console dmesg output. Because you can fix by putting in promiscuous mode, I suspect the problem is outside of the bridge itself, could be the drivers. The bridge itself puts the device into promiscuous mode already. Lastly, the Fedora and upstream kernels are not exactly the same. Please retest with a standard upstream kernel (like 2.6.39.2). ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 20:49 ` Stephen Hemminger @ 2011-07-11 21:08 ` Greg Scott 2011-07-11 21:10 ` Stephen Hemminger 2011-07-12 0:02 ` David Lamparter 0 siblings, 2 replies; 26+ messages in thread From: Greg Scott @ 2011-07-11 21:08 UTC (permalink / raw) To: Stephen Hemminger; +Cc: netdev, Lynn Hanson, Joe Whalen > What about console dmesg output. I should probably turn off all firewall logging so I don't fill the ring buffer with my log messages in, like, the first couple minutes after a boot. :) > Please retest with a standard upstream kernel (like 2.6.39.2). That's gonna take a while to put together a whole test environment with the latest kernel.org kernel. > The bridge itself puts the device into promiscuous mode already. Uhmmmm - no it didn't. Remember, I put br0 into promiscuous mode myself by hand - take a look at this. Note eth0 and eth1 are not in promiscuous mode. I wonder how it would behave if I put the physical devices into promiscuous mode and left br0 alone? This I can easily test during off hours. [root@ehac-fw2011 gregs]# ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000 link/ether 00:0e:7f:2d:d0:6e brd ff:ff:ff:ff:ff:ff 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0d:88:31:d8:24 brd ff:ff:ff:ff:ff:ff 5: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc prio state UNKNOWN link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff [root@ehac-fw2011 gregs]# - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 21:08 ` Greg Scott @ 2011-07-11 21:10 ` Stephen Hemminger 2011-07-11 21:16 ` Ben Greear 2011-07-11 21:16 ` Greg Scott 2011-07-12 0:02 ` David Lamparter 1 sibling, 2 replies; 26+ messages in thread From: Stephen Hemminger @ 2011-07-11 21:10 UTC (permalink / raw) To: Greg Scott; +Cc: netdev, Lynn Hanson, Joe Whalen On Mon, 11 Jul 2011 16:08:14 -0500 "Greg Scott" <GregScott@Infrasupport.com> wrote: > > What about console dmesg output. > > I should probably turn off all firewall logging so I don't fill the ring > buffer with my log messages in, like, the first couple minutes after a > boot. :) > > > Please retest with a standard upstream kernel (like 2.6.39.2). > > That's gonna take a while to put together a whole test environment with > the latest kernel.org kernel. > > > The bridge itself puts the device into promiscuous mode already. The bridge code calls dev_set_promiscuity() which should be changing device mode. But it could be that netdev core is resetting/changing/breaking that. ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 21:10 ` Stephen Hemminger @ 2011-07-11 21:16 ` Ben Greear 2011-07-12 3:06 ` Greg Scott 2011-07-11 21:16 ` Greg Scott 1 sibling, 1 reply; 26+ messages in thread From: Ben Greear @ 2011-07-11 21:16 UTC (permalink / raw) To: Stephen Hemminger; +Cc: Greg Scott, netdev, Lynn Hanson, Joe Whalen On 07/11/2011 02:10 PM, Stephen Hemminger wrote: > On Mon, 11 Jul 2011 16:08:14 -0500 > "Greg Scott"<GregScott@Infrasupport.com> wrote: > >>> What about console dmesg output. >> >> I should probably turn off all firewall logging so I don't fill the ring >> buffer with my log messages in, like, the first couple minutes after a >> boot. :) >> >>> Please retest with a standard upstream kernel (like 2.6.39.2). >> >> That's gonna take a while to put together a whole test environment with >> the latest kernel.org kernel. >> >>> The bridge itself puts the device into promiscuous mode already. > > The bridge code calls dev_set_promiscuity() which should > be changing device mode. But it could be that netdev core is > resetting/changing/breaking that. Last time I checked, 'ifconfig' and similar output didn't show promisc when NIC was actually promisc, unless the user specified the promisc-ness. You can read /sys/class/net/dev/eth0/flags and see if flag 0x100 is set..if so, it's promisc. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 21:16 ` Ben Greear @ 2011-07-12 3:06 ` Greg Scott 0 siblings, 0 replies; 26+ messages in thread From: Greg Scott @ 2011-07-12 3:06 UTC (permalink / raw) To: Ben Greear, Stephen Hemminger; +Cc: netdev, Lynn Hanson, Joe Whalen > You can read /sys/class/net/dev/eth0/flags and > see if flag 0x100 is set..if so, it's promisc. Hmmm... According to this, sure enough, all the relevant devices **are** set to PROMISC, even though ip link show doesn't say so. [root@ehac-fw2011 firewall-scripts]# more /sys/class/net/eth0/flags 0x1103 [root@ehac-fw2011 firewall-scripts]# more /sys/class/net/eth1/flags 0x1103 [root@ehac-fw2011 firewall-scripts]# more /sys/class/net/br0/flags 0x1103 [root@ehac-fw2011 firewall-scripts]# [root@ehac-fw2011 firewall-scripts]# [root@ehac-fw2011 firewall-scripts]# # eth2 is not part of the br0 bridge. [root@ehac-fw2011 firewall-scripts]# more /sys/class/net/eth2/flags 0x1003 [root@ehac-fw2011 firewall-scripts]# So the plot thickens - although I set br0 to PROMISC mode by hand here. Gathering similar data from an older and larger bridged site, this one running Fedora 9. Here I did not set br0 to PROMISC by hand, but it's set that way anyway. [root@lme-fw2 ~]# [root@lme-fw2 ~]# brctl showmacs br0 port no mac addr is local? ageing timer 2 00:00:aa:a9:c3:91 no 18.06 2 00:00:aa:ab:25:5b no 267.93 2 00:01:6c:68:9b:fc no 91.39 2 00:01:6c:68:9c:15 no 19.81 2 00:09:6b:83:30:fc no 48.57 2 00:0d:60:2b:cd:6b no 62.93 2 00:0d:60:70:bc:87 no 104.11 2 00:0e:0c:ee:5f:f0 no 1.27 1 00:11:21:bd:7f:80 no 0.03 2 00:11:25:6b:3a:8b no 39.18 2 00:12:79:7f:f5:2f no 8.66 2 00:14:38:de:4c:7a no 9.76 2 00:15:b7:22:43:ff no 1.30 2 00:15:b7:28:eb:fc no 232.10 2 00:15:b7:bf:bd:fc no 52.28 2 00:15:b7:c1:d8:fd no 118.31 2 00:15:b7:db:b0:fc no 275.73 2 00:15:f2:ea:93:c4 no 15.83 2 00:16:17:e0:24:72 no 158.90 2 00:17:31:46:e1:12 no 7.74 2 00:17:c5:14:e6:58 no 12.14 2 00:17:c5:14:e6:59 no 0.01 2 00:19:21:07:8e:8c no 37.98 2 00:19:db:a1:56:3e no 26.43 2 00:1b:78:22:63:0c no 19.87 2 00:1c:25:97:70:2e no 9.63 2 00:1c:c4:8e:dd:3e no 12.90 2 00:1e:0b:3e:7c:41 no 44.20 2 00:1e:0b:3e:7f:11 no 0.20 2 00:1e:4f:b9:2f:00 no 14.06 2 00:1f:29:14:f0:53 no 14.12 2 00:21:5e:89:92:d0 no 15.30 2 00:21:5e:b3:03:c1 no 0.17 2 00:22:68:59:b1:bf no 36.45 2 00:22:68:59:f5:a8 no 23.25 2 00:22:68:59:f7:0a no 106.53 2 00:22:68:59:fe:24 no 2.63 2 00:24:01:07:15:a6 no 2.12 2 00:26:5a:7b:0f:e3 no 161.61 2 00:30:80:d3:cd:20 no 1.54 2 00:40:af:81:7a:1b no 15.81 2 00:80:64:5e:5b:1b no 66.77 2 00:80:64:5e:5b:67 no 1.67 2 00:80:64:62:64:e8 no 2.12 2 00:90:dc:02:b5:9b no 70.30 2 00:90:dc:a4:0f:3b no 4.07 2 00:90:dc:a4:0f:3c no 22.30 2 00:a0:c8:20:54:05 no 0.23 2 00:c0:b6:0a:da:3c no 44.77 2 00:c0:b6:0a:da:3d no 44.77 2 10:78:d2:a8:e6:84 no 34.50 3 12:34:56:00:10:01 yes 0.00 2 12:34:56:00:10:11 no 1.02 2 12:34:56:00:30:05 yes 0.00 1 12:34:56:00:32:51 no 1.02 1 12:34:56:24:80:50 yes 0.00 2 40:61:86:7e:8c:93 no 10.02 2 6c:62:6d:5a:af:09 no 58.99 2 70:71:bc:51:8e:73 no 18.06 2 90:fb:a6:22:0a:06 no 23.24 2 e4:1f:13:2d:63:a8 no 0.31 [root@lme-fw2 ~]# [root@lme-fw2 ~]# [root@lme-fw2 ~]# brctl show bridge name bridge id STP enabled interfaces br0 8000.123456001001 no eth0 eth1 eth2 pan0 8000.000000000000 no [root@lme-fw2 ~]# more /sys/class/net/eth2/flags 0x1103 [root@lme-fw2 ~]# more /sys/class/net/eth1/flags 0x1103 [root@lme-fw2 ~]# more /sys/class/net/eth0/flags 0x1103 [root@lme-fw2 ~]# more /sys/class/net/br0/flags 0x1103 [root@lme-fw2 ~]# [root@lme-fw2 ~]# [root@lme-fw2 ~]# ip link show dev eth2 4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 12:34:56:00:10:01 brd ff:ff:ff:ff:ff:ff [root@lme-fw2 ~]# ip link show dev eth1 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 12:34:56:00:30:05 brd ff:ff:ff:ff:ff:ff [root@lme-fw2 ~]# ip link show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 12:34:56:24:80:50 brd ff:ff:ff:ff:ff:ff [root@lme-fw2 ~]# ip [root@lme-fw2 ~]# more /proc/version Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0 20080428 (Red H at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008 [link show dev br0 8: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue link/ether 12:34:56:00:10:01 brd ff:ff:ff:ff:ff:ff [root@lme-fw2 ~]# ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 21:10 ` Stephen Hemminger 2011-07-11 21:16 ` Ben Greear @ 2011-07-11 21:16 ` Greg Scott 2011-07-11 21:24 ` Stephen Hemminger 1 sibling, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-11 21:16 UTC (permalink / raw) To: Stephen Hemminger; +Cc: netdev, Lynn Hanson, Joe Whalen > The bridge code calls dev_set_promiscuity() which should > be changing device mode. But it could be that netdev core is > resetting/changing/breaking that. Is it supposed to change the physical ethnn devices or the br device? Here is what I do to set up the bridging. I do it myself right in the script so I can control all the details. . . . # # Setup bridging # echo "Setting up bridge $BR_IFACE to bridge $INET_IFACE with $TRUSTED1_IFACE" $BRCTL addbr $BR_IFACE $BRCTL addif $BR_IFACE $INET_IFACE $BRCTL addif $BR_IFACE $TRUSTED1_IFACE echo " Adding $BR_IP_SLASH and $TRUSTED1_IP_SLASH IP Addresses to $BR_IFACE" /sbin/ip addr add $BR_IP_SLASH broadcast $BR_BCAST_ADDRESS dev $BR_IFACE /sbin/ip addr add $TRUSTED1_IP_SLASH broadcast $TRUSTED1_BCAST_ADDRESS dev $BR_IFACE /sbin/ip link set $BR_IFACE up echo " Removing $INET_IP_SLASH and $TRUSTED1_IP_SLASH from $INET_IFACE and $TRUSTED1_IFACE" /sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE /sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE /sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE /sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE echo " Putting $BR_IFACE into promiscuous mode" # This fixes a bug forwarding packets bound for external IP Addresses # from the private LAN. ip link set $BR_IFACE promisc on # # Set up aliases for public IP addresses # . . . - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 21:16 ` Greg Scott @ 2011-07-11 21:24 ` Stephen Hemminger 0 siblings, 0 replies; 26+ messages in thread From: Stephen Hemminger @ 2011-07-11 21:24 UTC (permalink / raw) To: Greg Scott; +Cc: netdev, Lynn Hanson, Joe Whalen On Mon, 11 Jul 2011 16:16:40 -0500 "Greg Scott" <GregScott@Infrasupport.com> wrote: > > The bridge code calls dev_set_promiscuity() which should > > be changing device mode. But it could be that netdev core is > > resetting/changing/breaking that. > > Is it supposed to change the physical ethnn devices or the br device? The physical device ethnn. > Here is what I do to set up the bridging. I do it myself right in the > script so I can control all the details. > > . > . > . > # > # Setup bridging > # > > echo "Setting up bridge $BR_IFACE to bridge $INET_IFACE with > $TRUSTED1_IFACE" > > $BRCTL addbr $BR_IFACE > $BRCTL addif $BR_IFACE $INET_IFACE > $BRCTL addif $BR_IFACE $TRUSTED1_IFACE > > echo " Adding $BR_IP_SLASH and $TRUSTED1_IP_SLASH IP Addresses to > $BR_IFACE" > /sbin/ip addr add $BR_IP_SLASH broadcast $BR_BCAST_ADDRESS dev $BR_IFACE > /sbin/ip addr add $TRUSTED1_IP_SLASH broadcast $TRUSTED1_BCAST_ADDRESS > dev $BR_IFACE > /sbin/ip link set $BR_IFACE up > > echo " Removing $INET_IP_SLASH and $TRUSTED1_IP_SLASH from $INET_IFACE > and $TRUSTED1_IFACE" > /sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE > /sbin/ip addr del $INET_IP_SLASH dev $INET_IFACE > /sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE > /sbin/ip addr del $TRUSTED1_IP_SLASH dev $TRUSTED1_IFACE > > echo " Putting $BR_IFACE into promiscuous mode" > # This fixes a bug forwarding packets bound for external IP Addresses > # from the private LAN. > > ip link set $BR_IFACE promisc on > What is supposed to happen is that the bridge adds all the interface MAC addresses to the forwarding table as permanent entries. To show the forwarding table: # brctl showmacs br0 port no mac addr is local? ageing timer 1 c6:eb:2a:0c:12:eb yes 0.00 Then when packet arrives with that mac address it is handed up to netfilter, then if not firewalled, it goes on to the IP stack. There maybe protections against packet going back out the same interface that are getting tripped by all the rewriting. ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-11 21:08 ` Greg Scott 2011-07-11 21:10 ` Stephen Hemminger @ 2011-07-12 0:02 ` David Lamparter 2011-07-12 2:38 ` Greg Scott 1 sibling, 1 reply; 26+ messages in thread From: David Lamparter @ 2011-07-12 0:02 UTC (permalink / raw) To: Greg Scott; +Cc: Stephen Hemminger, netdev, Lynn Hanson, Joe Whalen On Mon, Jul 11, 2011 at 04:08:14PM -0500, Greg Scott wrote: > Uhmmmm - no it didn't. Remember, I put br0 into promiscuous mode myself > by hand - take a look at this. Note eth0 and eth1 are not in > promiscuous mode. I wonder how it would behave if I put the physical > devices into promiscuous mode and left br0 alone? This I can easily > test during off hours. > > [root@ehac-fw2011 gregs]# ip link show > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > 2: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state > DOWN qlen 1000 > link/ether 00:0e:7f:2d:d0:6e brd ff:ff:ff:ff:ff:ff > 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast > state UP qlen 1000 > link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff > 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast > state UP qlen 1000 > link/ether 00:0d:88:31:d8:24 brd ff:ff:ff:ff:ff:ff > 5: br0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc prio > state UNKNOWN > link/ether 00:03:47:3a:59:79 brd ff:ff:ff:ff:ff:ff > [root@ehac-fw2011 gregs]# All bridge port devices should really be in promiscuous mode; either this is just not displayed or the code is broken somewhere. On my 2.6.38 port both "ip" and "ifconfig" correctly show the devices as PROMISC without any userspace tool setting this, no idea if this changed recently. However, I must say that your bug report sounds more like a forwarding-back-to-source-device IP-level problem. If I understand your setup correctly, you have: (servers) | +--------[eth0 <-br0-> eth1]------- internet | (clients) and what isn't working is the firewall forwarding packets that it received on eth0/br0 back out on eth0/br0 on the IP level including NATting? Or did I misunderstand? -David P.S.: any reason why you are (a) not using proxyarp instead of the bridge and (b) not using a VLAN to put those servers with public IPs in? You have a bit of a Frankennet there :/ ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-12 0:02 ` David Lamparter @ 2011-07-12 2:38 ` Greg Scott 2011-07-12 3:39 ` David Lamparter 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-12 2:38 UTC (permalink / raw) To: David Lamparter; +Cc: Stephen Hemminger, netdev, Lynn Hanson, Joe Whalen > If I understand your setup correctly, you have: > > (servers) > | > +--------[eth0 <-br0-> eth1]------- internet > | > (clients) Close. Here's a better ASCII art picture. There aren't really any internal-->external clients. Internet firewall Private LAN ----------------+******+---------+-------+ eth0 eth1 NATed H.323 Bridge br0 servers devices The H.323 devices work better if they have real, public IP Addresses. I've done them with NAT, but H.323 just works better if the devices "think" they're directly connected to the Internet. All the servers are all on the physical private LAN side, physically behind the firewall. > Why not proxy ARP? I used to use proxy ARP until I got burned really badly with what proxy ARP really does - the NIC answers ARP requests (in proxy) for everyone and anyone that asks with its own MAC address. Think about that - proxy ARP impersonates everyone and anyone on the LAN to which it's connected. I had one of these in a colo center and for several hours, my box Proxy-ARPed everyone and anyone on that same public network. I don't even like to think about how many public webservers I unintentionally messed with that day. Oh yes - and to make matters worse, that customer had an IP load balancer behind my box nobody told me about and proxy ARP messed that up too. The stupid load balancer wouldn't clear its ARP cache and had to be rebooted - and that took down a major website and pretty much blew my only chance to do business with this customer. All in all, not one of my better days. I decided right then and there, no more proxy ARP. Bridging turns out to be a much cleaner and more polite way to do it. Don't believe all the forum comments about the wonders of proxy-ARP. > Why not use a VLAN? Because I really don't need one. Plus it doesn't matter anyway - the firewall can act as a router on a stick to go between my H.323 devices and private IP servers. With or without VLANs makes no difference in this case. > You have a bit of a Frankennet there I don't think so. I have a single LAN with a couple of devices that need public IP Addresses. This isn't that unusual. I have lots of other sites doing it this way. > I must say that your bug report sounds more like a > forwarding-back-to-source-device IP-level problem. I don't think it's an IP level problem. I think it's a layer 2 problem - and now I think the problem is, bridging is supposed to turn on PROMISC mode and it didn't. I had to do it by hand myself. I never paid attention to whether or not PROMISC mode was turned on with any of the other ones like this I've done and never had a problem with it until this one. And as soon as I turned on PROMISC mode by hand, everything worked as it should. If it was an IP problem, or routing problem, or ruleset/filtering problem, why would PROMISC mode make any difference one way or the other? What I don't know yet is, is this a Fedora bug or a stock kernel bug? Is anyone from Red Hat following this email list? I think I will take a look at a few of my other bridged sites running earlier versions and see if they turn on PROMISC mode on their bridged NICs. - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-12 2:38 ` Greg Scott @ 2011-07-12 3:39 ` David Lamparter 2011-07-12 14:30 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: David Lamparter @ 2011-07-12 3:39 UTC (permalink / raw) To: Greg Scott Cc: David Lamparter, Stephen Hemminger, netdev, Lynn Hanson, Joe Whalen Hi -- First of all, I still can't find your kernel version in any of your mails. Can you please repeat the uname -a output of the affected box? Anyway, On Mon, Jul 11, 2011 at 09:38:51PM -0500, Greg Scott wrote: > > Why not proxy ARP? > > I used to use proxy ARP until I got burned really badly with what proxy > ARP really does - the NIC answers ARP requests (in proxy) for everyone > and anyone that asks with its own MAC address. Think about that - proxy > ARP impersonates everyone and anyone on the LAN to which it's connected. Yes. That's why you use ip neighbor add proxy 1.2.3.4 dev eth0 (for that to work, you need forwarding on, and the route for the host must point to a different outbound device) I agree that the proxy_arp /proc knob is pure evil. In my opinion it should be removed in favour of "ip neighbor add proxy". > > Why not use a VLAN? > > Because I really don't need one. Plus it doesn't matter anyway - the > firewall can act as a router on a stick to go between my H.323 devices > and private IP servers. With or without VLANs makes no difference in > this case. The VLAN saves you the SNAT on your clients traffic towards the NATed services, because the traffic back from those NATed services goes through the firewall, which will apply its conntrack entries. Also, what you're doing is a case of _layer 3_ routing of packets that arrive at an interface - br0 - back out to the same interface - br0. I'm not excluding the possibility that the bridge code is broken, but my nose tells me that your SNAT-DNAT setup is more likely to have gotten broken. Either way I still don't understand your setup. Are you using ebtables? Where is your private IP that's facing towards the clients? Is there a separate third DMZ network? What is $DMZ_IFACE? (Posting your setup script is a noble thought completeness-wise, but tbfh it's unreadable and maybe even the wrong parts. Please boil it down to a few lines and please include the client direction...) > - and now I think the problem is, bridging is supposed to turn on > PROMISC mode and it didn't. I had to do it by hand myself. So it works when you switch the bridge members into PROMISC? (not the bridge itself!) > What I don't know yet is, is this a Fedora bug or a stock kernel bug? > Is anyone from Red Hat following this email list? I only know I haven't seen it anywhere, but my bridges are getting fewer and fewer... -David ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-12 3:39 ` David Lamparter @ 2011-07-12 14:30 ` Greg Scott 2011-07-12 14:54 ` David Lamparter 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-12 14:30 UTC (permalink / raw) To: David Lamparter; +Cc: Stephen Hemminger, netdev, Lynn Hanson, Joe Whalen > First of all, I still can't find your kernel version in any of your > mails. Can you please repeat the uname -a output of the affected box? Woops - sorry - I never posted it, that's why you didn't see it. Here it is: [root@ehac-fw2011 firewall-scripts]# uname -a Linux ehac-fw2011 2.6.35.6-48.fc14.i686.PAE #1 SMP Fri Oct 22 15:27:53 UTC 2010 i686 i686 i386 GNU/Linux It's just as Red Hat delivered it. > The VLAN saves you the SNAT on your clients traffic towards the NATed > services, because the traffic back from those NATed services goes > through the firewall, which will apply its conntrack entries. I don't see it that way. I have a couple of devices with public IP Addresses and most with "normal" private IP Addresses. Those public IP devices can easily be on the same Ethernet segment and in the same collision domain as the private ones. There's no good reason to separate them at this particular site. Oh - I think see what you're thinking - the words, "public IP Address", lead to a wrong conclusion that those devices really are **public**. Just because some of the devices have public IP Addresses does **NOT** mean they're completely accessible to the public. Just like for, say, web servers, we NAT TCP port 80 and block the rest - for these public IP Address devices at this site, I ACCEPT TCP 1720 and use the H.323 conntrack module to handle that traffic because H.323 does not get along easily with NAT. Pretty much the only difference between the public IP stuff and the private IP stuff is, I NAT for the private IP stuff and just ACCEPT traffic I want to let in/out for the public IP stuff. There's no reason to separate the public IP stuff and private IP stuff with VLANs. Anyway, at this site, the H.323 stuff is the **reason** why I need a bridge, so the H.323 stuff is only indirectly related to the problem. I have other sites with different reasons for various systems to have real public IP Addresses. > Also, what you're doing is a case of _layer 3_ routing of packets that > arrive at an interface - br0 - back out to the same interface - br0. Yes, absolutely, when internal users need to access the NATed websites using public IP Addresses instead of their private IP Addresses. Classic router on a stick topology, but using DNAT and MASQUERADE. Let me try to describe it this way. Forget about the reason I need a bridge. I have a good reason this site is bridged and have now hopefully presented a reasonable case why I need one. I have a public website at private IP Address 192.168.10.2. The firewall DNATs TCP port 80 for public IP Address aa.bb.115.151 to 192.168.10.2. Most of this traffic will come in from the Internet. But I also want my **internal** users to have the same experience as the rest of the world when viewing this website, so that traffic will come in from the private LAN. I have good reasons for this - one biggie is so my internal users can see website changes as the rest of the world sees them. I know I can simulate this with a private DNS server, but I don't like multiple versions of DNS floating around. So I choose to do it with layer 3 routing and some fancy DNAT and SNAT (really MASQUERADing) at the firewall. This all worked for several years, right up until I put in a replacement based on Fedora 14. By now, the script I have in place at this site is battle hardened and has been in service for close to 10 years and many hardware upgrades. And it broke with Fedora 14. > Either way I still don't understand your setup. Are you using ebtables? Yes to ebtables. I use ebtables to mark packets so I know which interface they come in on. Anything coming in from the Internet gets a mark of 1. Anything coming in on the LAN side gets a mark of 2. You have me kind of afraid to post the code....but I'll paste it in below anyway. btw, I posted code earlier because one reply asked me to do so. . . . echo "Flushing and zeroing all ebtables tables and chains" $EBTABLES -t broute -F $EBTABLES -t broute -Z $EBTABLES -t filter -F $EBTABLES -t filter -Z $EBTABLES -t nat -F $EBTABLES -t nat -Z # # Use ebtables to mark packets based on the in/out interface. # 1 - (bit 0 set) for packets entering on the Internet physical interface # 2 - (bit 1 set) for packets entering on the trusted physical interface # 3 - (bits 0 and 1) for packets exiting via the Internet physical interface # (Kernel 2.6.23 or so changed the order of iptables/ebtables going out, so # marking outbound packets is meaningless now.) echo "Marking bridged packets at layer 2 for later layer 3 filtering." $EBTABLES -t broute -A BROUTING -i $INET_IFACE \ -j mark --mark-set 1 --mark-target CONTINUE $EBTABLES -t broute -A BROUTING -i $TRUSTED1_IFACE \ -j mark --mark-set 2 --mark-target CONTINUE . . . > Is there a separate third DMZ network? What is $DMZ_IFACE? That DMZ network is not relevant and I should not have included any reference to it. It's eth2 and the NIC is in the box but nothing is connected to it. And it's not part of the bridge. It's an empty NIC that we may use in the future, but right now not relevant. > Where is your private IP that's facing towards the clients? I don't know what this question means. The setup is a traditional Public<-->firewall<-->private topology, as the ASCII art I posted earlier shows. But some of the stuff on the private side needs public IP Addresses, so the firewall is a bridge plus a router, not just a router. > So it works when you switch the bridge members into PROMISC? (not the > bridge itself!) No, the br0 bridge device itself. After a bunch of troubleshooting, below is literally the single one and only change I needed to make this work again. . . . echo " Putting $BR_IFACE into promiscuous mode" # This fixes a bug forwarding packets bound for external IP Addresses # from the private LAN. ip link set $BR_IFACE promisc on . . . I don't think I should need to do this by hand and I never needed it before. That's why it took me weeks and plenty of help with the Netfilter folks to find it. Something apparently changed with bridging. Reading through some of the replies to this post, I decided to look and see what happens with the physical ethnnn devices when I add them to a bridge, so I looked at a couple of other sites with similar setups. I've never had any need to dig into this before because it all just worked. - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bridging behavior apparently changed around the Fedora 14 time 2011-07-12 14:30 ` Greg Scott @ 2011-07-12 14:54 ` David Lamparter 2011-07-12 16:28 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: David Lamparter @ 2011-07-12 14:54 UTC (permalink / raw) To: Greg Scott; +Cc: David Lamparter, netdev, Lynn Hanson, Joe Whalen On Tue, Jul 12, 2011 at 09:30:05AM -0500, Greg Scott wrote: > Linux ehac-fw2011 2.6.35.6-48.fc14.i686.PAE #1 SMP Fri Oct 22 15:27:53 > UTC 2010 i686 i686 i386 GNU/Linux > > It's just as Red Hat delivered it. Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but I think you need to go bug RedHat. Anyway, either someone else should have had your problem by now (so it might be fixed) or I'd say there's something wrong with your setup (maybe changed defaults) or RedHat messed up the kernel ;). > > The VLAN saves you the SNAT on your clients traffic towards the NATed > > services, because the traffic back from those NATed services goes > > through the firewall, which will apply its conntrack entries. > > I don't see it that way. I have a couple of devices with public IP > Addresses and most with "normal" private IP Addresses. Those public IP [snip] You totally misunderstood me. I'm suggesting the separate VLAN for your servers which have private IPs but which have services exposed to the internet (and your clients) on public IPs through NAT. Your H323 stuff is totally unrelated. > > Also, what you're doing is a case of _layer 3_ routing of packets that > > arrive at an interface - br0 - back out to the same interface - br0. > > Yes, absolutely, when internal users need to access the NATed websites > using public IP Addresses instead of their private IP Addresses. > Classic router on a stick topology, but using DNAT and MASQUERADE. Where's the log fire for your router on a stick? *SCNR* > Let me try to describe it this way. Forget about the reason I need a > bridge. I have a good reason this site is bridged and have now > hopefully presented a reasonable case why I need one. Yes. Your problem seems to be between the private-IP clients in your network and your private-IP servers if I understand correctly. The bridge is most likely completely innocent and your "stick" NAT setup just broke down due to some changed default I'd guess. > And it broke with Fedora 14. > > Where is your private IP that's facing towards the clients? > > I don't know what this question means. The setup is a traditional > Public<-->firewall<-->private topology, as the ASCII art I posted > earlier shows. But some of the stuff on the private side needs public > IP Addresses, so the firewall is a bridge plus a router, not just a > router. Yes. And because it is a router, it as an IP from the private subnet your clients are in. My question was: what device is that IP on? > > So it works when you switch the bridge members into PROMISC? (not the > > bridge itself!) > > No, the br0 bridge device itself. After a bunch of troubleshooting, > below is literally the single one and only change I needed to make this > work again. This makes me think yet more that the bridge code is innocent. > I don't think I should need to do this by hand and I never needed it > before. That's why it took me weeks and plenty of help with the > Netfilter folks to find it. Something apparently changed with bridging. No. You're jumping to conclusions. You're affecting the "top" bridge device's promiscuity. I would say that the effect you're seeing is in the IP stack above it, caused by it now promiscuously handling packets that are dropped otherwise. Setups like yours need a lot of caution regarding ICMP redirect / shared_media / ARP settings. Please check those. -David P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :) ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-12 14:54 ` David Lamparter @ 2011-07-12 16:28 ` Greg Scott 2011-07-21 4:40 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-12 16:28 UTC (permalink / raw) To: David Lamparter; +Cc: netdev, Lynn Hanson, Joe Whalen > P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :) Sorry - didn't ignore it, just didn't reply back to it. I'll look into it. What I've read about this before has all been kind of vague. Does this mean I proxy ARP only for IP Address 1.2.3.4? So somebody sends an ARP whois 1.2.3.4, I'll answer with 1.2.3.4. is at {My MAC Address}? If so, then I agree, not nearly as evil as just setting proxy_arp. > Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but I > think you need to go bug RedHat. Yeah, maybe. OK, probably. This was such a bizarre problem - I started with Netfilter and those guys suggested I try here. At least now I understand the problem lots better than before. And it's not like I can just go and update dozens of kernels at dozens of sites all the time when a new kernel comes out. > You totally misunderstood me. I'm suggesting the separate VLAN for your > servers which have private IPs but which have services exposed to the > internet (and your clients) on public IPs through NAT. Ahh - OK. The challenge with many small sites is, economic reality. That same server that hosts the public ftp and websites also hosts all the internal Windows file/print services. It's the only server at this site, so it has several roles. I would love to build a real DMZ network and put all the public facing stuff in there, but I don't have money for multiple servers. This will become even more difficult to separate when we go to virtual servers and clustered hosts. > Your H323 stuff is totally unrelated. Agreed. Wholeheartedly. > Yes. Your problem seems to be between the private-IP clients in your > network and your private-IP servers if I understand correctly. Yes. Dead-bang, right on target. > Yes. And because it is a router, it as an IP from the private subnet > your clients are in. My question was: what device is that IP on? Ahh - eth1 is the private LAN side, 192.168.10.1. All the NATed LAN stuff and all the workstations are in the 192.168.10.0/24 subnet and connected to eth1. Eth0 is the Internet side. The Internet side has the firewall NIC, a cable, and the Internet router. That's it. Everything is connected to the LAN side. > No. You're jumping to conclusions. You're affecting the "top" bridge > device's promiscuity. I would say that the effect you're seeing is in > the IP stack above it, caused by it now promiscuously handling packets > that are dropped otherwise. Well they were sure dropped before I set it to PROMISC mode, that's for sure. And it all worked with the earlier version. That's why this feels like a layer 2 issue. If it was an IP issue, why didn't it break several years ago when I first set it up? Does bridging make everything a little more complex and delicate to set up? Well, yeah. And some of the netfilter stuff has been a moving target over the years. I don't see how ICMP redirects matter. Comparing /proc/sys/net/ipv4/conf/*/accept_redirects with this version and an older one at another site - all identical. ../all/accept_recdirects is 0, the rest are all 1. Shared media and ARP settings - /proc/sys/net/ipv4/conf/*/shared_media - all 1 for all interfaces. There are a zillion arp settings. Looking at /proc/sys/net/ipv4/conf/*/*arp* - all are 0 in both the other older site and this newer site. Curiously - at one of my other older sites, apparently br0 is not in promisc mode. But I don't think these guys do any of the stick routing stuff. I wonder if these guys have the problem but we don't see it because they never try it? [root@NSSSS-fw1 ~]# more /sys/class/net/br0/flags 0x1003 [root@NSSSS-fw1 ~]# [root@NSSSS-fw1 ~]# more /proc/version Linux version 2.6.32.11-99.fc12.i686.PAE (mockbuild@x86-05.phx2.fedoraproject.org) (gcc version 4.4.3 20100127 (Red Hat 4.4.3-4) (GCC) ) #1 SMP Mon Apr 5 16:15:03 EDT 2010 [root@NSSSS-fw1 ~]# [root@NSSSS-fw1 ~]# uname -a Linux NSSSS-fw1 2.6.32.11-99.fc12.i686.PAE #1 SMP Mon Apr 5 16:15:03 EDT 2010 i686 i686 i386 GNU/Linux [root@NSSSS-fw1 ~]# Here is a much older bridged site based on Fedora 9 and I'm sure these guys use my stick routing stuff. Look at the difference in ..br0/flags. [root@lme-fw2 ~]# more /sys/class/net/br0/flags 0x1103 [root@lme-fw2 ~]# [root@lme-fw2 ~]# more /proc/version Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0 20080428 (Red H at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008 [root@lme-fw2 ~]# [root@lme-fw2 ~]# uname -a Linux lme-fw2 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686 i686 i386 GNU/Linux I can still get my hands on the old box at the site in question. I guess it couldn't hurt to fire it up and look at its br0 flags. - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-12 16:28 ` Greg Scott @ 2011-07-21 4:40 ` Greg Scott 2011-07-21 15:01 ` Greg Scott [not found] ` <925A849792280C4E80C5461017A4B8A2A0413A@mail733.InfraSupportE tc.com> 0 siblings, 2 replies; 26+ messages in thread From: Greg Scott @ 2011-07-21 4:40 UTC (permalink / raw) To: Greg Scott, David Lamparter Cc: netdev, Lynn Hanson, Joe Whalen, Graham Parenteau Aw nuts, nothing is ever straightforward. When I do: ip link set br0 promisc on My internal users can see the internally hosted websites using the public IP Addresses. The router on a stick rules I put in work just fine. (In on br0/eth1, DNATed in PREROUTING, MASQUERADEd in POSTROUTING, back out br0/eth1 to the correct internal host.) However, I just learned tonight, this breaks both inbound and outbound PPTP VPNs. And when I do: ip link set br0 promisc off now my PPTP VPNs work, but this breaks my above router on a stick rules. My PPTP VPN stuff uses the GRE iptables conntrack modules, ip_conntrack_pptp and ip_nat_pptp, and some PREROUTING and POSTROUTING rules to DNAT TCP 1723 and all GRE packets to an internal Windows RRAS server. But when I turn promisc on for br0, I see a storm of packets looping over and over again, until the remote client finally times out after what seems like an eternity. I'll bet that bridge forwards my packets out the wrong physical ethnn interface when it's in promisc mode and that's why my NATed PPTP VPN breaks. Which makes me wonder if putting br0 in promisc mode breaks any of my other NATed services. - Greg Scott -----Original Message----- From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Greg Scott Sent: Tuesday, July 12, 2011 11:29 AM To: David Lamparter Cc: netdev@vger.kernel.org; Lynn Hanson; Joe Whalen Subject: RE: Bridging behavior apparently changed around the Fedora 14 time > P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :) Sorry - didn't ignore it, just didn't reply back to it. I'll look into it. What I've read about this before has all been kind of vague. Does this mean I proxy ARP only for IP Address 1.2.3.4? So somebody sends an ARP whois 1.2.3.4, I'll answer with 1.2.3.4. is at {My MAC Address}? If so, then I agree, not nearly as evil as just setting proxy_arp. > Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but I > think you need to go bug RedHat. Yeah, maybe. OK, probably. This was such a bizarre problem - I started with Netfilter and those guys suggested I try here. At least now I understand the problem lots better than before. And it's not like I can just go and update dozens of kernels at dozens of sites all the time when a new kernel comes out. > You totally misunderstood me. I'm suggesting the separate VLAN for your > servers which have private IPs but which have services exposed to the > internet (and your clients) on public IPs through NAT. Ahh - OK. The challenge with many small sites is, economic reality. That same server that hosts the public ftp and websites also hosts all the internal Windows file/print services. It's the only server at this site, so it has several roles. I would love to build a real DMZ network and put all the public facing stuff in there, but I don't have money for multiple servers. This will become even more difficult to separate when we go to virtual servers and clustered hosts. > Your H323 stuff is totally unrelated. Agreed. Wholeheartedly. > Yes. Your problem seems to be between the private-IP clients in your > network and your private-IP servers if I understand correctly. Yes. Dead-bang, right on target. > Yes. And because it is a router, it as an IP from the private subnet > your clients are in. My question was: what device is that IP on? Ahh - eth1 is the private LAN side, 192.168.10.1. All the NATed LAN stuff and all the workstations are in the 192.168.10.0/24 subnet and connected to eth1. Eth0 is the Internet side. The Internet side has the firewall NIC, a cable, and the Internet router. That's it. Everything is connected to the LAN side. > No. You're jumping to conclusions. You're affecting the "top" bridge > device's promiscuity. I would say that the effect you're seeing is in > the IP stack above it, caused by it now promiscuously handling packets > that are dropped otherwise. Well they were sure dropped before I set it to PROMISC mode, that's for sure. And it all worked with the earlier version. That's why this feels like a layer 2 issue. If it was an IP issue, why didn't it break several years ago when I first set it up? Does bridging make everything a little more complex and delicate to set up? Well, yeah. And some of the netfilter stuff has been a moving target over the years. I don't see how ICMP redirects matter. Comparing /proc/sys/net/ipv4/conf/*/accept_redirects with this version and an older one at another site - all identical. ../all/accept_recdirects is 0, the rest are all 1. Shared media and ARP settings - /proc/sys/net/ipv4/conf/*/shared_media - all 1 for all interfaces. There are a zillion arp settings. Looking at /proc/sys/net/ipv4/conf/*/*arp* - all are 0 in both the other older site and this newer site. Curiously - at one of my other older sites, apparently br0 is not in promisc mode. But I don't think these guys do any of the stick routing stuff. I wonder if these guys have the problem but we don't see it because they never try it? [root@NSSSS-fw1 ~]# more /sys/class/net/br0/flags 0x1003 [root@NSSSS-fw1 ~]# [root@NSSSS-fw1 ~]# more /proc/version Linux version 2.6.32.11-99.fc12.i686.PAE (mockbuild@x86-05.phx2.fedoraproject.org) (gcc version 4.4.3 20100127 (Red Hat 4.4.3-4) (GCC) ) #1 SMP Mon Apr 5 16:15:03 EDT 2010 [root@NSSSS-fw1 ~]# [root@NSSSS-fw1 ~]# uname -a Linux NSSSS-fw1 2.6.32.11-99.fc12.i686.PAE #1 SMP Mon Apr 5 16:15:03 EDT 2010 i686 i686 i386 GNU/Linux [root@NSSSS-fw1 ~]# Here is a much older bridged site based on Fedora 9 and I'm sure these guys use my stick routing stuff. Look at the difference in ..br0/flags. [root@lme-fw2 ~]# more /sys/class/net/br0/flags 0x1103 [root@lme-fw2 ~]# [root@lme-fw2 ~]# more /proc/version Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0 20080428 (Red H at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008 [root@lme-fw2 ~]# [root@lme-fw2 ~]# uname -a Linux lme-fw2 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686 i686 i386 GNU/Linux I can still get my hands on the old box at the site in question. I guess it couldn't hurt to fire it up and look at its br0 flags. - Greg -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-21 4:40 ` Greg Scott @ 2011-07-21 15:01 ` Greg Scott [not found] ` <925A849792280C4E80C5461017A4B8A2A0413A@mail733.InfraSupportE tc.com> 1 sibling, 0 replies; 26+ messages in thread From: Greg Scott @ 2011-07-21 15:01 UTC (permalink / raw) To: netdev; +Cc: Lynn Hanson, Joe Whalen, Graham Parenteau, David Lamparter Aw nuts, nothing is ever straightforward. When I do: ip link set br0 promisc on My internal users can see the internally hosted websites using the public IP Addresses. The router on a stick rules I put in work just fine. (In on br0/eth1, DNATed in PREROUTING, MASQUERADEd in POSTROUTING, back out br0/eth1 to the correct internal host.) However, I just learned last night, this breaks both inbound and outbound PPTP VPNs. And when I do: ip link set br0 promisc off now my PPTP VPNs work, but this breaks my above router on a stick rules. My PPTP VPN stuff uses the GRE iptables conntrack modules, ip_conntrack_pptp and ip_nat_pptp, and some PREROUTING and POSTROUTING rules to DNAT TCP 1723 and all GRE packets to an internal Windows RRAS server. But when I turn promisc on for br0, I see a storm of packets looping over and over again, until the remote client finally times out after what seems like an eternity. I'll bet that bridge forwards my packets out the wrong physical ethnn interface when it's in promisc mode and that's why my NATed PPTP VPN breaks. Which makes me wonder if putting br0 in promisc mode breaks any of my other NATed services. So back to square one I guess. - Greg Scott -----Original Message----- From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On Behalf Of Greg Scott Sent: Tuesday, July 12, 2011 11:29 AM To: David Lamparter Cc: netdev@vger.kernel.org; Lynn Hanson; Joe Whalen Subject: RE: Bridging behavior apparently changed around the Fedora 14 time > P.S.: you blissfully ignored my "ip neigh add proxy 1.2.3.4" note :) Sorry - didn't ignore it, just didn't reply back to it. I'll look into it. What I've read about this before has all been kind of vague. Does this mean I proxy ARP only for IP Address 1.2.3.4? So somebody sends an ARP whois 1.2.3.4, I'll answer with 1.2.3.4. is at {My MAC Address}? If so, then I agree, not nearly as evil as just setting proxy_arp. > Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but I > think you need to go bug RedHat. Yeah, maybe. OK, probably. This was such a bizarre problem - I started with Netfilter and those guys suggested I try here. At least now I understand the problem lots better than before. And it's not like I can just go and update dozens of kernels at dozens of sites all the time when a new kernel comes out. > You totally misunderstood me. I'm suggesting the separate VLAN for your > servers which have private IPs but which have services exposed to the > internet (and your clients) on public IPs through NAT. Ahh - OK. The challenge with many small sites is, economic reality. That same server that hosts the public ftp and websites also hosts all the internal Windows file/print services. It's the only server at this site, so it has several roles. I would love to build a real DMZ network and put all the public facing stuff in there, but I don't have money for multiple servers. This will become even more difficult to separate when we go to virtual servers and clustered hosts. > Your H323 stuff is totally unrelated. Agreed. Wholeheartedly. > Yes. Your problem seems to be between the private-IP clients in your > network and your private-IP servers if I understand correctly. Yes. Dead-bang, right on target. > Yes. And because it is a router, it as an IP from the private subnet > your clients are in. My question was: what device is that IP on? Ahh - eth1 is the private LAN side, 192.168.10.1. All the NATed LAN stuff and all the workstations are in the 192.168.10.0/24 subnet and connected to eth1. Eth0 is the Internet side. The Internet side has the firewall NIC, a cable, and the Internet router. That's it. Everything is connected to the LAN side. > No. You're jumping to conclusions. You're affecting the "top" bridge > device's promiscuity. I would say that the effect you're seeing is in > the IP stack above it, caused by it now promiscuously handling packets > that are dropped otherwise. Well they were sure dropped before I set it to PROMISC mode, that's for sure. And it all worked with the earlier version. That's why this feels like a layer 2 issue. If it was an IP issue, why didn't it break several years ago when I first set it up? Does bridging make everything a little more complex and delicate to set up? Well, yeah. And some of the netfilter stuff has been a moving target over the years. I don't see how ICMP redirects matter. Comparing /proc/sys/net/ipv4/conf/*/accept_redirects with this version and an older one at another site - all identical. ../all/accept_recdirects is 0, the rest are all 1. Shared media and ARP settings - /proc/sys/net/ipv4/conf/*/shared_media - all 1 for all interfaces. There are a zillion arp settings. Looking at /proc/sys/net/ipv4/conf/*/*arp* - all are 0 in both the other older site and this newer site. Curiously - at one of my other older sites, apparently br0 is not in promisc mode. But I don't think these guys do any of the stick routing stuff. I wonder if these guys have the problem but we don't see it because they never try it? [root@NSSSS-fw1 ~]# more /sys/class/net/br0/flags 0x1003 [root@NSSSS-fw1 ~]# [root@NSSSS-fw1 ~]# more /proc/version Linux version 2.6.32.11-99.fc12.i686.PAE (mockbuild@x86-05.phx2.fedoraproject.org) (gcc version 4.4.3 20100127 (Red Hat 4.4.3-4) (GCC) ) #1 SMP Mon Apr 5 16:15:03 EDT 2010 [root@NSSSS-fw1 ~]# [root@NSSSS-fw1 ~]# uname -a Linux NSSSS-fw1 2.6.32.11-99.fc12.i686.PAE #1 SMP Mon Apr 5 16:15:03 EDT 2010 i686 i686 i386 GNU/Linux [root@NSSSS-fw1 ~]# Here is a much older bridged site based on Fedora 9 and I'm sure these guys use my stick routing stuff. Look at the difference in ..br0/flags. [root@lme-fw2 ~]# more /sys/class/net/br0/flags 0x1103 [root@lme-fw2 ~]# [root@lme-fw2 ~]# more /proc/version Linux version 2.6.25-14.fc9.i686 (mockbuild@) (gcc version 4.3.0 20080428 (Red H at 4.3.0-8) (GCC) ) #1 SMP Thu May 1 06:28:41 EDT 2008 [root@lme-fw2 ~]# [root@lme-fw2 ~]# uname -a Linux lme-fw2 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686 i686 i386 GNU/Linux I can still get my hands on the old box at the site in question. I guess it couldn't hurt to fire it up and look at its br0 flags. - Greg -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <925A849792280C4E80C5461017A4B8A2A0413A@mail733.InfraSupportE tc.com>]
* RE: Bridging behavior apparently changed around the Fedora 14 time [not found] ` <925A849792280C4E80C5461017A4B8A2A0413A@mail733.InfraSupportE tc.com> @ 2011-07-22 4:39 ` Greg Scott 2011-07-22 6:20 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-22 4:39 UTC (permalink / raw) To: netdev; +Cc: Lynn Hanson, Joe Whalen, Graham Parenteau, David Lamparter > Whoa. And here I was almost ashamed of running 2.6.38. I'm sorry, but I > think you need to go bug RedHat. Per your suggestion, I just put together a Red Hat bugzilla report and documented the behavior to the best of my ability. It's listed as bug #724862 in bugzilla.redhat.com. - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Bridging behavior apparently changed around the Fedora 14 time 2011-07-22 4:39 ` Greg Scott @ 2011-07-22 6:20 ` Greg Scott 2011-09-15 22:48 ` Very confused about broute DROP Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-07-22 6:20 UTC (permalink / raw) To: netdev; +Cc: Lynn Hanson, Joe Whalen, Graham Parenteau, David Lamparter Hang on a second - David, I think you're trying to tell me I'm the problem. Maybe so... I send a frame in on eth1/br0. It traverses a bunch of ebtables and iptables DNAT and SNAT rules and then wants to come back out on eth1/br0. But this is both a bridge and a router, and trying to remember my bridge basics - as I recall, bridges might not want to forward frames back out the same physical interface they came in on. After running this frame/packet through a gauntlet of ebtables/iptables rules and NATing the snot out of the IP packet inside the frame, I wonder if it's still the same layer 2 frame when it gets ready to come back out on eth1? Even though NAT changed the layer 3 routed packet IP info, I wonder if it's still the same layer 2 datalink frame? So the bridge says, "No way Jose, that's the same physical interface you came in on" and drops it. And then turning on promisc mode makes the bridge less picky and it happily forwards all frames everywhere. Is this what you've been trying to tell me and I'm too thickheaded to get it? If so, to work around the problem, I was reading about some ebtables broute ACCEPT/DROP targets that are apparently named badly - DROP apparently tells the bridge to route the packet instead of bridging the frame, while ACCEPT tells the bridge to bridge it. Should I be looking in this direction? But thinking about this some more - there are cases, such as cascaded switches, or hubs behind a switch, where you want a bridge to forward frames back out the same interface. Well . . . maybe. There's still something about forwarding out the same interface a frame comes in on that's bugging me. - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
* Very confused about broute DROP 2011-07-22 6:20 ` Greg Scott @ 2011-09-15 22:48 ` Greg Scott 2011-09-15 23:08 ` Christian Benvenuti (benve) 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-09-15 22:48 UTC (permalink / raw) To: netdev; +Cc: Graham Parenteau I don't get this. Why does: ebtables -t broute -A BROUTING -j DROP completely knock a Linux host offline? This is what the man page for ebtables says: The targets DROP and ACCEPT have a special meaning in the broute table (these names are used instead of more descriptive names to keep the implementation generic). DROP actually means the frame has to be routed, while ACCEPT means the frame has to be bridged. The BROUTING chain is traversed very early. However, it is only traversed by frames entering on a bridge port that is in forwarding state. Normally those frames would be bridged, but you can decide otherwise here. The redirect target is very handy here. So based on the above paragraph, I should be able to do something like this: # Here is what to bridge ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP1 -j ACCEPT ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP2 -j ACCEPT # Route everything else ebtables -t broute -A BROUTING -j DROP So I tried above and knocked that box completely offline. I'm missing something. Here is what the paragraph about redirect in the ebtables man pages says: The redirect target will change the MAC target address to that of the bridge device the frame arrived on. This target can only be used in the BROUTING chain of the broute table and the PREROUTING chain of the nat table. In the BROUTING chain, the MAC address of the bridge port is used as destination address, in the PREROUTING chain, the MAC address of the bridge is used. OK - so this target MAC address - is this the MAC Address of an ethnn port that's part of the bridge, or the MAC Address of another node? I was thinking it was the MAC Address of another node, but maybe it's just the MAC Address of a port on this bridge? And there are some examples here: http://ebtables.sourceforge.net/examples/basic.html#ex_redirect that I really don't get. So instead of trial and error guessing, I figured I would ask. If anyone can help me understand this, I'll take a stab at writing it up as clearly as I know how for use in future versions of man pages. Thanks - Greg Scott ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Very confused about broute DROP 2011-09-15 22:48 ` Very confused about broute DROP Greg Scott @ 2011-09-15 23:08 ` Christian Benvenuti (benve) 2011-09-16 3:19 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Christian Benvenuti (benve) @ 2011-09-15 23:08 UTC (permalink / raw) To: Greg Scott, netdev; +Cc: Graham Parenteau How about ARP? You need it too ... > -----Original Message----- > From: netdev-owner@vger.kernel.org [mailto:netdev- > owner@vger.kernel.org] On Behalf Of Greg Scott > Sent: Thursday, September 15, 2011 3:48 PM > To: netdev@vger.kernel.org > Cc: Graham Parenteau > Subject: Very confused about broute DROP > > I don't get this. Why does: > > ebtables -t broute -A BROUTING -j DROP > > completely knock a Linux host offline? > > This is what the man page for ebtables says: > > The targets DROP and ACCEPT have a special meaning in the broute table > (these names are used instead of more descriptive names to keep the > implementation generic). DROP actually means the frame has to be > routed, while ACCEPT means the frame has to be bridged. The BROUTING > chain is traversed very early. However, it is only traversed by > frames entering on a bridge port that is in forwarding state. > Normally those frames would be bridged, but you can decide otherwise > here. The redirect target is very handy here. > > So based on the above paragraph, I should be able to do something like > this: > > # Here is what to bridge > ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP1 -j > ACCEPT > ebtables -t broute -A BROUTING -p IPv4 --ip-destination $PUBLIC_IP2 -j > ACCEPT > > # Route everything else > ebtables -t broute -A BROUTING -j DROP > > So I tried above and knocked that box completely offline. I'm missing > something. > > Here is what the paragraph about redirect in the ebtables man pages > says: > > The redirect target will change the MAC target address to that of the > bridge device the frame arrived on. This target can only be used in the > BROUTING chain of the broute table and the PREROUTING chain of the > nat > table. In the BROUTING chain, the MAC address of the bridge port > is > used as destination address, in the PREROUTING chain, the MAC address > of > the bridge is used. > > OK - so this target MAC address - is this the MAC Address of an ethnn > port that's part of the bridge, or the MAC Address of another node? I > was thinking it was the MAC Address of another node, but maybe it's > just > the MAC Address of a port on this bridge? > > And there are some examples here: > http://ebtables.sourceforge.net/examples/basic.html#ex_redirect > > that I really don't get. So instead of trial and error guessing, I > figured I would ask. > > If anyone can help me understand this, I'll take a stab at writing it > up > as clearly as I know how for use in future versions of man pages. > > Thanks > > - Greg Scott > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Very confused about broute DROP 2011-09-15 23:08 ` Christian Benvenuti (benve) @ 2011-09-16 3:19 ` Greg Scott 2011-09-16 4:23 ` Christian Benvenuti (benve) 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-09-16 3:19 UTC (permalink / raw) To: Christian Benvenuti (benve), netdev; +Cc: Graham Parenteau If I'm reading the ebtables ARP stuff right, it looks like I can use this for selective proxy ARPs. This may have made my life much nicer with a project that didn't work out very well 5 years ago when I inherited a network with systems that had public IP Addresses on both sides of the firewall. For now, why did ebtables -t broute -A BROUTING -j DROP shut my system down and force me to drive over there at 4AM this morning to undo it? It's not supposed to really drop, it's supposed to route it. And those examples aren't making any sense to me. Thanks - Greg -----Original Message----- From: Christian Benvenuti (benve) [mailto:benve@cisco.com] Sent: Thursday, September 15, 2011 6:08 PM To: Greg Scott; netdev@vger.kernel.org Cc: Graham Parenteau Subject: RE: Very confused about broute DROP How about ARP? You need it too ... ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Very confused about broute DROP 2011-09-16 3:19 ` Greg Scott @ 2011-09-16 4:23 ` Christian Benvenuti (benve) 2011-09-16 14:55 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Christian Benvenuti (benve) @ 2011-09-16 4:23 UTC (permalink / raw) To: Greg Scott, netdev; +Cc: Graham Parenteau What I meant is that your host needs to be able to route (which means ... to process) its own ARP traffic, ... IPv4 does not work without ARP, right? This means you need to add one more DROP rule for the ARP traffic that is addressed to the MAC of the host interfaces (nothing to do with proxy ARP). /Chris > -----Original Message----- > From: Greg Scott [mailto:GregScott@Infrasupport.com] > Sent: Thursday, September 15, 2011 8:19 PM > To: Christian Benvenuti (benve); netdev@vger.kernel.org > Cc: Graham Parenteau > Subject: RE: Very confused about broute DROP > > If I'm reading the ebtables ARP stuff right, it looks like I can use > this for selective proxy ARPs. This may have made my life much nicer > with a project that didn't work out very well 5 years ago when I > inherited a network with systems that had public IP Addresses on both > sides of the firewall. For now, why did > > ebtables -t broute -A BROUTING -j DROP > > shut my system down and force me to drive over there at 4AM this > morning > to undo it? It's not supposed to really drop, it's supposed to route > it. And those examples aren't making any sense to me. > > Thanks > > - Greg > > > > -----Original Message----- > From: Christian Benvenuti (benve) [mailto:benve@cisco.com] > Sent: Thursday, September 15, 2011 6:08 PM > To: Greg Scott; netdev@vger.kernel.org > Cc: Graham Parenteau > Subject: RE: Very confused about broute DROP > > How about ARP? You need it too ... > ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Very confused about broute DROP 2011-09-16 4:23 ` Christian Benvenuti (benve) @ 2011-09-16 14:55 ` Greg Scott 2011-09-18 1:47 ` Greg Scott 0 siblings, 1 reply; 26+ messages in thread From: Greg Scott @ 2011-09-16 14:55 UTC (permalink / raw) To: Christian Benvenuti (benve), netdev; +Cc: Graham Parenteau Definitely counter-intuitive in my head. Three little ebtables rules - ACCEPT anything IPv4 - which includes ARPs - for a couple of IP Addresses and then DROP everything else for all protocols. (Which really means bridge for those IPv4 Addresses and route for everything else.) I don't see how a rule specific to ARPs matters here. ARP is an IPv4 protocol and should be implied in the rules I set up - right? Why a rule specific to ARPs when it's already part of IPv4? This just hit me - layer 2 is stateless. I have ACCEPT rules for frames bound for **destination** IPv4 IP Addresses. I wonder if I need similar rules for these same IP Addresses as sources? But that still doesn't make sense - that rule to DROP everything else covers all sources and all destinations. When I put in that ebtables DROP rule, the box turns into a black hole instead of a router. Maybe one day the lightbulb will light up in my head, but right now I still don't get it. - Greg -----Original Message----- From: Christian Benvenuti (benve) [mailto:benve@cisco.com] Sent: Thursday, September 15, 2011 11:23 PM To: Greg Scott; netdev@vger.kernel.org Cc: Graham Parenteau Subject: RE: Very confused about broute DROP What I meant is that your host needs to be able to route (which means ... to process) its own ARP traffic, ... IPv4 does not work without ARP, right? This means you need to add one more DROP rule for the ARP traffic that is addressed to the MAC of the host interfaces (nothing to do with proxy ARP). /Chris ^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: Very confused about broute DROP 2011-09-16 14:55 ` Greg Scott @ 2011-09-18 1:47 ` Greg Scott 0 siblings, 0 replies; 26+ messages in thread From: Greg Scott @ 2011-09-18 1:47 UTC (permalink / raw) To: Christian Benvenuti (benve), netdev; +Cc: Graham Parenteau A clarification - hopefully my embarrassing mistake can be of some use. ARP is an L2 protocol, not an L3 protocol. ARP is **not** part of IPv4 or IPv6. Look in the IANA list of IP protocol numbers - there's none for ARP. ARP is defined in Layer 2, not L3. I would have missed that question on a cert test. That's why my broute rules need to take ARPs into account. - Greg ^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2011-09-18 1:47 UTC | newest] Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-07-11 18:25 Bridging behavior apparently changed around the Fedora 14 time Greg Scott 2011-07-11 20:07 ` Stephen Hemminger 2011-07-11 20:41 ` Greg Scott 2011-07-11 20:49 ` Stephen Hemminger 2011-07-11 21:08 ` Greg Scott 2011-07-11 21:10 ` Stephen Hemminger 2011-07-11 21:16 ` Ben Greear 2011-07-12 3:06 ` Greg Scott 2011-07-11 21:16 ` Greg Scott 2011-07-11 21:24 ` Stephen Hemminger 2011-07-12 0:02 ` David Lamparter 2011-07-12 2:38 ` Greg Scott 2011-07-12 3:39 ` David Lamparter 2011-07-12 14:30 ` Greg Scott 2011-07-12 14:54 ` David Lamparter 2011-07-12 16:28 ` Greg Scott 2011-07-21 4:40 ` Greg Scott 2011-07-21 15:01 ` Greg Scott [not found] ` <925A849792280C4E80C5461017A4B8A2A0413A@mail733.InfraSupportE tc.com> 2011-07-22 4:39 ` Greg Scott 2011-07-22 6:20 ` Greg Scott 2011-09-15 22:48 ` Very confused about broute DROP Greg Scott 2011-09-15 23:08 ` Christian Benvenuti (benve) 2011-09-16 3:19 ` Greg Scott 2011-09-16 4:23 ` Christian Benvenuti (benve) 2011-09-16 14:55 ` Greg Scott 2011-09-18 1:47 ` Greg Scott
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).