RE: Ebtables hook order anomaly

RE: Ebtables hook order anomaly

[Greg Scott]

> Thats indeed inconsistent. I don't believe this has changed however, 
> the IPv4 POSTROUTING hook was always called from the bridge 
> POSTROUTING hook (with similar priorities).

Well, OK - so I guess the most important question now is, are there any
other behavior changes of this nature coming up so I can get ready for
them in advance?  I'm kind of hoping for no more rude midnight
surprises.  :)

Thanks

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> Thats indeed inconsistent. I don't believe this has changed however, 
>> the IPv4 POSTROUTING hook was always called from the bridge 
>> POSTROUTING hook (with similar priorities).
> 
> Well, OK - so I guess the most important question now is, are there any
> other behavior changes of this nature coming up so I can get ready for
> them in advance?  I'm kind of hoping for no more rude midnight
> surprises.  :)

Looking at feature-removal-schedule, not from my side :)

RE: Ebtables hook order anomaly

[Greg Scott]

> Well maybe you don't want to use a bridge for simplicity 
> but still do proxy arp for lots of addresses.

See the email I sent a few minutes ago about proxy ARP.  Proxy ARP
scares the daylights out of me.  Now, if there were a way to proxy ARP
for only a select set of IP Addresses - instead of ***all*** IP
Addresses hitting the interface - now that would be something!

- Greg

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Wednesday 2008-04-09 17:04, Patrick McHardy wrote:
>> 
>>  - arpreply engine
>
> A pure bridge doesn't need ARP. I don't get your point,
> but my suggestion might not make much sense, I haven't
> really thought about this.

Well maybe you don't want to use a bridge for simplicity
but still do proxy arp for lots of addresses.

>>  - filtering on L2
>> 
>> there has been a L2-hooks back in the 2.4 days (at least the context
>> looks like it), and given that the ebtables targets now use
>> xtables, the l2hooks patch would actually be real easy.
>
> They were used for the kernel ct_sync implementation. Why
> would you use them for briding, it has its own hooks?
>
If you wanted to filter out non-(arp,ipv4,ipv6) traffic,
you have to set up a bridge and filter it there, as I see it.
So L2 hooks would come in handy since you would not need
to set up a bridge device anymore to do so.

RE: Ebtables hook order anomaly

[Greg Scott]

>>> I forgot why exactly you need the bridge port in iptables.
>> 
>> We crossed in the email a minute ago when I described my H.323 stuff.


> Yes, but I was wondering more about the exact rules where
> this would be helpful. The things you described in your
> other mail sounds easily doable by simply exluding port
> 1723 from NAT without any knowledge of bridge ports.

I also need to know the direction - is the packet going back inside or
out to the Internet.  Same for input - is it coming from the inside or
coming in from the Internet.  Input is more important than output - but
knowing the output direction would be nice.  

Examples - 

I have a section in iptables-NAT-PREROUTING that tests on input for
anything from 10.n/8 or 172.16/12 or 192.168/16.  If in from the
Internet than I log and drop it.  Without knowing the in-interface, I
don't know how I would do those tests.  

Here is a forward rule for outbound stuff that broke with the recent
change:

$IPTABLES -A FORWARD -m mark --mark 3 -j ACCEPT
# Packets marked with 3 (bits 0 and 1 set) are outbound to the Internet.

The mark was for anything from the inside going out to the Internet.  

The FORWARD rules are not always that simple.  I have some sites that
need to regulate what can go out.  With the new ebtables order, I can
deal with that with IP Addresses instead of eth interfaces.  But it
would still be nice to have the ability to regulate by interface instead
of IP Address.  

Here was another one that made me scratch my bald head most of the night
before Easter - when this rule broke, you can imagine the weird
behavior:

echo "	MASQUERADing everything else going outbound"
##$IPTABLES -t nat -A POSTROUTING -m mark --mark 3 -j LOG --log-prefix
"mark3 "
$IPTABLES -t nat -A POSTROUTING -m mark --mark 3 -j MASQUERADE

I can change this to test for any IP Address other than the local LAN
and it should work.  It just makes things a little more complicated for
sites that have many local LANs.

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> I forgot why exactly you need the bridge port in iptables.
> 
> We crossed in the email a minute ago when I described my H.323 stuff.  

Yes, but I was wondering more about the exact rules where
this would be helpful. The things you described in your
other mail sounds easily doable by simply exluding port
1723 from NAT without any knowledge of bridge ports.

RE: Ebtables hook order anomaly

[Greg Scott]

> I forgot why exactly you need the bridge port in iptables.

We crossed in the email a minute ago when I described my H.323 stuff.  

There was another case a year or so ago that first turned me on to
bridging.  I inherited a network that had systems with IP Addresses all
over the place both in front and behind the firewall.  It was mess.  

I tried to set up a Linux firewall with proxy ARP but the results were
very ugly.  It turned out, they had this load balancer in their network
nobody told me about and my proxy ARP and the load-balancer's proxy ARPs
got confused with each other and this took down a popular website for an
hour or so at 5 in the morning central time.  

And then I finally figured out what proxy ARP really meant when my
outside eth0 interface was proxy-ARPing.  This was in a co-lo site and
there were other customers on the same Ethernet and I ended up
proxy-ARPing for them - woops!  This didn't cause any damage but it sure
scared me to death!

Needless to say, that was the last time I ever used proxy-ARP.  

So now I use bridging whenever I have devices that need public IP
Addresses inside a mostly NATed network.

- Greg

RE: Ebtables hook order anomaly

[Greg Scott]

> A pure bridge doesn't need ARP. I don't get your point,
> but my suggestion might not make much sense, I haven't
> really thought about this.

My little bridge-router systems could use ARP.  I am putting together a
site right now that uses an active/passive HA pair of firewall
bridge/router systems.  I kludge in a MAC Address for the active system
to make the router in front of it happy, as well as sending out a GARP.


This site needs bridging because it has H.323 videoconferencing
equipment in place.  H.323 devices - especially older ones - do not get
along well with NAT.  So I give the H.323 stuff a public IP Address and
set up a bridge.  

This all worked nicely until the night before Easter when I tried a new
version and everything broke.  But I can live with not knowing the out
interface from L3 as long as I know the in-interface.  

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> there has been a L2-hooks back in the 2.4 days (at least the context
>> looks like it), and given that the ebtables targets now use
>> xtables, the l2hooks patch would actually be real easy.
> 
> All I ask is, please please please don't break the order with ebtables
> and iptables on input!  I really really really need to know the in-eth
> interface.  

Don't worry.

> Also, I'm willing to test any l2-hook patches that tell me the iptables
> out eth interface in a bridge.  I have a couple of sites where I can
> brew up kernels and rulesets.

That won't work. iptables *can't* know since it sits at the network
layer, while briding sits *below* it at the device layer.

I forgot why exactly you need the bridge port in iptables. But
in any case the only way to get it is within briding (where some
of the iptables features are not available). So a fix for all
of this should be to make the missing iptables features available
for briding natively.

RE: Ebtables hook order anomaly

[Greg Scott]

> there has been a L2-hooks back in the 2.4 days (at least the context
> looks like it), and given that the ebtables targets now use
> xtables, the l2hooks patch would actually be real easy.

All I ask is, please please please don't break the order with ebtables
and iptables on input!  I really really really need to know the in-eth
interface.  

Also, I'm willing to test any l2-hook patches that tell me the iptables
out eth interface in a bridge.  I have a couple of sites where I can
brew up kernels and rulesets.

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> On Wednesday 2008-04-09 16:36, Patrick McHardy wrote:
>> Jan Engelhardt wrote:
>>> It explains things, but having the feature removed is not nice.
>>> I cannot deny that the previous code was real a hook spaghetteria,
>>> but how could it be done better?
>> Good question. The order on output is logically correct, so I
>> wouldn't change it. I never liked the invocation of IP hooks
>> from bridging at all, so long-term we could consider making
>> the features people want to bridging available "natively"
>> (like conntrack/NAT/...).
> 
> Can't quite imagine what you mean. Bridging is currently
> implemented using a separate netdevice, much like bonding
> or gre/ipip/tun/tap tunnels.
> 
> Well bridging is essentially a few pieces:
> 
>  - "forward" packets without changing MAC
> 
>  - arpreply engine

A pure bridge doesn't need ARP. I don't get your point,
but my suggestion might not make much sense, I haven't
really thought about this.

> I have often come across needing a bridge device with a single port and
> brouting=drop only because of ebt_arpreply; I have immediate plans
> to remove this limitation.
> 
>  - STP
> 
> I seem to remember ideas about a userspace daemon were floating
> around..
> 
>  - filtering on L2
> 
> there has been a L2-hooks back in the 2.4 days (at least the context
> looks like it), and given that the ebtables targets now use
> xtables, the l2hooks patch would actually be real easy.

They were used for the kernel ct_sync implementation. Why
would you use them for briding, it has its own hooks?

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Wednesday 2008-04-09 16:36, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> On Tuesday 2008-03-25 13:57, Patrick McHardy wrote:
>> > > > which means ebtables actually comes after iptables, and hence,
>> > > > your mark 3 will not show up as you expected.
>> > Indeed, on output bridge netfilter will run after IPv4 netfilter.
>> > Does that explain things?
>> 
>> It explains things, but having the feature removed is not nice.
>> I cannot deny that the previous code was real a hook spaghetteria,
>> but how could it be done better?
>
> Good question. The order on output is logically correct, so I
> wouldn't change it. I never liked the invocation of IP hooks
> from bridging at all, so long-term we could consider making
> the features people want to bridging available "natively"
> (like conntrack/NAT/...).

Can't quite imagine what you mean. Bridging is currently
implemented using a separate netdevice, much like bonding
or gre/ipip/tun/tap tunnels.

Well bridging is essentially a few pieces:

 - "forward" packets without changing MAC

 - arpreply engine

I have often come across needing a bridge device with a single port and
brouting=drop only because of ebt_arpreply; I have immediate plans
to remove this limitation.

 - STP

I seem to remember ideas about a userspace daemon were floating
around..

 - filtering on L2

there has been a L2-hooks back in the 2.4 days (at least the context
looks like it), and given that the ebtables targets now use
xtables, the l2hooks patch would actually be real easy.

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> On Tuesday 2008-03-25 13:57, Patrick McHardy wrote:
>>>> which means ebtables actually comes after iptables, and hence,
>>>> your mark 3 will not show up as you expected.
>> Indeed, on output bridge netfilter will run after IPv4 netfilter.
>> Does that explain things?
> 
> It explains things, but having the feature removed is not nice.
> I cannot deny that the previous code was real a hook spaghetteria,
> but how could it be done better?

Good question. The order on output is logically correct, so I
wouldn't change it. I never liked the invocation of IP hooks
from bridging at all, so long-term we could consider making
the features people want to bridging available "natively"
(like conntrack/NAT/...).

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Tuesday 2008-03-25 13:57, Patrick McHardy wrote:
>> >
>> > which means ebtables actually comes after iptables, and hence,
>> > your mark 3 will not show up as you expected.
>
> Indeed, on output bridge netfilter will run after IPv4 netfilter.
> Does that explain things?

It explains things, but having the feature removed is not nice.
I cannot deny that the previous code was real a hook spaghetteria,
but how could it be done better?

RE: Ebtables hook order anomaly

[Greg Scott]

> This still seems to work - ebtables BROUTE still seems to come 
> before iptables NAT PREROUTING and my ebtables BROUTE marks all 
> show up in iptables.  Am I on solid ground for the future if I 
> keep this up?

So can I count on this working for the long term?

ie - Can I mark packets in ebtables-BROUTE based on the in-interface and
then read the mark later on in iptables?  (I already know I can't do
this any more for the out-interface.)

Thanks

- Greg

RE: Ebtables hook order anomaly

[Greg Scott]

>>> and POSTROUTING hook until the outgoing bridge port was determined
by 
>>> the bridge code. This "feature" was removed because it broke all 
>>> kinds of things, now the order matches the layering and IPv4 hooks 
>>> are always processed entirely before bridging.
>> 
>> Now the order is .. non-consistent.
>> On a pure bridge forward (-i br -o br), as I have determined, 
>> ebtables-nat-POSTROUTING comes _before_ the IPv4 hooks.
>
> Thats indeed inconsistent. I don't believe this has changed however, 
> the IPv4 POSTROUTING hook was always called from the bridge
POSTROUTING 
> hook (with similar priorities).

This woke me up in the middle of the night - I also mark packets in
ebtables BROUTE based on the incoming interface and then test all over
the place in iptables based on that mark.  One of the most important is
a test for -s {private IP Address} coming in from the Internet.  But
there are lots of other tests based on source IP and incoming interface.
I really really really need to know the incoming interface.  

This still seems to work - ebtables BROUTE still seems to come before
iptables NAT PREROUTING and my ebtables BROUTE marks all show up in
iptables.  Am I on solid ground for the future if I keep this up?

Thanks

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> 
> On Tuesday 2008-03-25 14:49, Patrick McHardy wrote:
>> Greg Scott wrote:
>>>
>>>  It could be something in the order of execution changed.  I'm using
>>>  RedHat kernels right now and I know they tweak the kernels a little 
>>> bit.
>>>  But surely the RedHat guys would not change something this fundamental?
>>
>> No, that was us :) Bridge-netfilter used to defer the IPv4 OUTPUT
> 
> Do you have the commit id at hand? Was it
> 2bf540b73ed5b304e84bb4d4c390d49d1cfa0ef8?

Yes.

>> and POSTROUTING hook until the outgoing bridge port was determined
>> by the bridge code. This "feature" was removed because it broke
>> all kinds of things, now the order matches the layering and IPv4
>> hooks are always processed entirely before bridging.
> 
> Now the order is .. non-consistent.
> On a pure bridge forward (-i br -o br), as I have determined,
> ebtables-nat-POSTROUTING comes _before_ the IPv4 hooks.

Thats indeed inconsistent. I don't believe this has changed
however, the IPv4 POSTROUTING hook was always called from
the bridge POSTROUTING hook (with similar priorities).

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Tuesday 2008-03-25 14:49, Patrick McHardy wrote:
> Greg Scott wrote:
>>
>>  It could be something in the order of execution changed.  I'm using
>>  RedHat kernels right now and I know they tweak the kernels a little bit.
>>  But surely the RedHat guys would not change something this fundamental?
>
> No, that was us :) Bridge-netfilter used to defer the IPv4 OUTPUT

Do you have the commit id at hand? Was it
2bf540b73ed5b304e84bb4d4c390d49d1cfa0ef8?

> and POSTROUTING hook until the outgoing bridge port was determined
> by the bridge code. This "feature" was removed because it broke
> all kinds of things, now the order matches the layering and IPv4
> hooks are always processed entirely before bridging.

Now the order is .. non-consistent.
On a pure bridge forward (-i br -o br), as I have determined,
ebtables-nat-POSTROUTING comes _before_ the IPv4 hooks.

RE: Ebtables hook order anomaly

[Greg Scott]

> For routed packets you can't make any decisions in iptables based on 
> the outgoing bridge port, thats only possible for purely bridged 
> traffic.

Bummer - but at least I now know the rules.  I guess I'll test using
source IP Address.  That kind of works - I know what stuff is on either
side - but it's messy.  

Thanks

- GReg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> Indeed, on output bridge netfilter will run after IPv4 netfilter.
>> Does that explain things?
> 
> Well, not really.  2.6.23-2 behaves differently than 2.6.18.  My
> ebtables marks get set and read by iptables in 2.6.18.  iptables does
> not see them in 2.6.23.
> 
> It could be something in the order of execution changed.  I'm using
> RedHat kernels right now and I know they tweak the kernels a little bit.
> But surely the RedHat guys would not change something this fundamental?


No, that was us :) Bridge-netfilter used to defer the IPv4 OUTPUT
and POSTROUTING hook until the outgoing bridge port was determined
by the bridge code. This "feature" was removed because it broke
all kinds of things, now the order matches the layering and IPv4
hooks are always processed entirely before bridging.

> I set my ebtables mark for outbound packets in two places - in the
> OUTPUT chain and the FORWARD chain.  I was looking mostly at the
> ebtables FORWARD chain for packets that pass thru the box.  I set the
> mark in ebtables FORWARD and OUTPUT.  Iptables POSTROUTING needs to see
> them so it knows to MASQUERADE.  Otherwise, things get really weird.  :)
> iptables sees the marks in 2.6.18, but does not see them in 2.6.23.  
> 
> I have a couple of 2.6.23 systems here and I can brew up a 2.6.18 fc6
> system later on for testing if needed.  (All the 2.6.18 stuff I have
> built up right now is in production.)  I can set up some tests similarly
> to Jan's testing from last night for packets passing through the box.  
> 
> Or if there's a better way to do this, I'm open.  

For routed packets you can't make any decisions in iptables based
on the outgoing bridge port, thats only possible for purely bridged
traffic.

RE: Ebtables hook order anomaly

[Greg Scott]

> Indeed, on output bridge netfilter will run after IPv4 netfilter.
> Does that explain things?

Well, not really.  2.6.23-2 behaves differently than 2.6.18.  My
ebtables marks get set and read by iptables in 2.6.18.  iptables does
not see them in 2.6.23.

It could be something in the order of execution changed.  I'm using
RedHat kernels right now and I know they tweak the kernels a little bit.
But surely the RedHat guys would not change something this fundamental?


I set my ebtables mark for outbound packets in two places - in the
OUTPUT chain and the FORWARD chain.  I was looking mostly at the
ebtables FORWARD chain for packets that pass thru the box.  I set the
mark in ebtables FORWARD and OUTPUT.  Iptables POSTROUTING needs to see
them so it knows to MASQUERADE.  Otherwise, things get really weird.  :)
iptables sees the marks in 2.6.18, but does not see them in 2.6.23.  

I have a couple of 2.6.23 systems here and I can brew up a 2.6.18 fc6
system later on for testing if needed.  (All the 2.6.18 stuff I have
built up right now is in production.)  I can set up some tests similarly
to Jan's testing from last night for packets passing through the box.  

Or if there's a better way to do this, I'm open.  

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> 
> On Tuesday 2008-03-25 03:28, Jan Engelhardt wrote:
>>
>>>  None of my packets that were supposed to have that "3" mark ever kept
>>>  them.  For some reason, they either never were marked or they were
>>>  marked and then the mark disappeared.
>>
>> I can reproduce it. [...]
>> Now that is indeed interesting. I had this thought, maybe the mark 
>> does not disappear, maybe ebtables is run -- contrary to most graphics 
>> depicting the netfilter flow -- _after_ iptables. So I tried:
>>
>>      iptables -t mangle -A POSTROUTING -j LOG --log-prefix "[ipt] " -d
>>      134.76.13.21
>>      ebtables -A OUTPUT -p ipv4 --ip-destination 134.76.13.21 --log
>>      --log-prefix "[ebt] "
>>
>> with the result of:
>>
>>      [214961.190130] [ipt] IN= OUT=br0 SRC=10.10.106.161 DST=134.76.13.21
>>      LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0
>>      ID=59252 SEQ=1
>>
>>      [214961.190186] [ebt]  IN= OUT=sis0 MAC source = 00:0a:e6:98:ed:d7
>>      MAC
>>      dest = 68:a8:3e:d3:d0:fb proto = 0x0800
>>
>> which means ebtables actually comes after iptables, and hence, your 
>> mark 3 will not show up as you expected.


Indeed, on output bridge netfilter will run after IPv4 netfilter.
Does that explain things?