Feature suggestion ...

Feature suggestion ...

[Akshat Kakkar]

Hi All,

With latest addition of storing skbinfo (mainly skbprio) in ipset and
then applying it later to the traffic as and when it passes the
iptables ruleset, it becomes relatively easy and simple to do traffic
shaping.

IMHO what one of the feature set which we can add is to have an ipset
of only fwmarks i.e. fw-marks will be hashed and stored in the ipset
and then later using these marks we can instantly (in single look up)
set traffic class.
Motivation for this is the fact that just reading the mark can signify
which fw rule it belongs to (no matter how complex was that firewall
rule). So if we do traffic shaping based on marks, it becomes
straightforward traffic shaping for that fw rule. Now as no of rules
increase, this lookup will also increase linearly and
more-importantly, we cant take advantage of any statefulness over
here, i.e. if there are 100 rules then everytime my packet will have
to go down 100 lines, get it matched, get its skbprio value and then
move forward.

however, if we such an ipset as I am mentioning, so all these skbprio
settings and all can always be done in single shot,

say we make something like
ipset -N MARK-TC-MAP hash:mark skbinfo

iptables -t mangle -A POSTROUTING -j SET --map-set MARK-TC-MAP src --map-prio


Whats your view on this?

Re: Feature suggestion ...

[Jozsef Kadlecsik]

On Sun, 6 Sep 2015, Akshat Kakkar wrote:

> With latest addition of storing skbinfo (mainly skbprio) in ipset and 
> then applying it later to the traffic as and when it passes the iptables 
> ruleset, it becomes relatively easy and simple to do traffic shaping.
> 
> IMHO what one of the feature set which we can add is to have an ipset of 
> only fwmarks i.e. fw-marks will be hashed and stored in the ipset and 
> then later using these marks we can instantly (in single look up) set 
> traffic class.

You can already store skbmark in the set element extension and set the 
mark value to the matching packets. 

> Motivation for this is the fact that just reading the mark can signify
> which fw rule it belongs to (no matter how complex was that firewall
> rule). So if we do traffic shaping based on marks, it becomes
> straightforward traffic shaping for that fw rule. Now as no of rules
> increase, this lookup will also increase linearly and
> more-importantly, we cant take advantage of any statefulness over
> here, i.e. if there are 100 rules then everytime my packet will have
> to go down 100 lines, get it matched, get its skbprio value and then
> move forward.

Sorry, but I don't undestand this...
 
> however, if we such an ipset as I am mentioning, so all these skbprio
> settings and all can always be done in single shot,
> 
> say we make something like
> ipset -N MARK-TC-MAP hash:mark skbinfo
> 
> iptables -t mangle -A POSTROUTING -j SET --map-set MARK-TC-MAP src --map-prio

...and your example doesn't help either. Why do you need the set at all?
You could simply write

iptables -t mangle -A POSTROUTING -j MARK --set-mark value

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: Feature suggestion ...

[Akshat Kakkar]

Web
more
Sent Mail
Feature suggestion ...
InboxNETFILTER

Akshat Kakkar
Hi All, With latest addition of storing skbinfo (mainly skbprio) in
ipset and then applying it later to the traffic as and when it passes
the iptab...
J
Jozsef Kadlecsik
You can already store skbmark in the set element extension and set the
mark value to the matching packets. Sorry, but I don't undestand
this... ......

Akshat Kakkar
to Jozsef, netfilter
1 minute agoDetails
I am suggesting an ipset hash:mark.

Let me explain the motivation for this requirement:

Assume we have 100 fw rules each marking packet as 1 to 100. I am
marking these to do traffic shaping, so that I need not check fw
matching conditions on every packet. Simple check on mark will be
sufficient.
iptables -t mangle  -A Forward -j mark --restore-mark
iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept

iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
condition 1> -j MARK --set-mark 1

iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
condition 2> -j MARK --set-mark 2
.
.
.
iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
condition 100> -j MARK --set-mark 100

iptables -t mangle -A Forward -j Connmark --save-mark

Next would be Filter table in Forward chain:

iptables -t filter -m connmark ! --mark 0 -j Accept

Note that as we are using connmark so we don't require related,
established rule.

Now as I have to do bw shaping, so I need 100 tc filter rules, something like

tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1

tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 2 fw flowid 1:2

tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 3 fw flowid 1:3
.
.
.
tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
(pardon me if I am wrong on syntax, idea is to give a feel of things)

Now visualize traffic for rule 100. every pkt, in tc, will face a
delay equal 100T where T is the time to search first entry, as search
will be linear. Clearly this doesn't scale well when rule count moves
to thousand or more.

However, if we have an ipset hash:mark with skbinfo support, then we
can store this mark - tc_class membership in it and then with a
constant lookup time we can scale to any no. of rules with cost being
only memory:

ipset -N mark_tc_class_map hash:mark skbinfo

ipset -A mark_tc_class_map 1 skbprio 1:1
ipset -A mark_tc_class_map 2 skbprio 1:2
ipset -A mark_tc_class_map 3 skbprio 1:3
.
.
.
ipset -A mark_tc_class_map 4 skbprio 1:100

Please note that this is not storing mark in skbinfo but creating hash
of marks and then storing skbinfo against each mark.

This ipset then we will use in mangle chain of postrouting

iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
With above rule we don't require those 100 tc filters mentioned above.
It all reduces to single rule in iptables and constant lookup time for
traffic shaping.



On 9/7/15, Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> wrote:
> On Sun, 6 Sep 2015, Akshat Kakkar wrote:
>
>> With latest addition of storing skbinfo (mainly skbprio) in ipset and
>> then applying it later to the traffic as and when it passes the iptables
>> ruleset, it becomes relatively easy and simple to do traffic shaping.
>>
>> IMHO what one of the feature set which we can add is to have an ipset of
>> only fwmarks i.e. fw-marks will be hashed and stored in the ipset and
>> then later using these marks we can instantly (in single look up) set
>> traffic class.
>
> You can already store skbmark in the set element extension and set the
> mark value to the matching packets.
>
>> Motivation for this is the fact that just reading the mark can signify
>> which fw rule it belongs to (no matter how complex was that firewall
>> rule). So if we do traffic shaping based on marks, it becomes
>> straightforward traffic shaping for that fw rule. Now as no of rules
>> increase, this lookup will also increase linearly and
>> more-importantly, we cant take advantage of any statefulness over
>> here, i.e. if there are 100 rules then everytime my packet will have
>> to go down 100 lines, get it matched, get its skbprio value and then
>> move forward.
>
> Sorry, but I don't undestand this...
>
>> however, if we such an ipset as I am mentioning, so all these skbprio
>> settings and all can always be done in single shot,
>>
>> say we make something like
>> ipset -N MARK-TC-MAP hash:mark skbinfo
>>
>> iptables -t mangle -A POSTROUTING -j SET --map-set MARK-TC-MAP src
>> --map-prio
>
> ...and your example doesn't help either. Why do you need the set at all?
> You could simply write
>
> iptables -t mangle -A POSTROUTING -j MARK --set-mark value
>
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary
>

Re: Feature suggestion ...

[Akshat Kakkar]

I am suggesting an ipset hash:mark.

Let me explain the motivation for this requirement:

Assume we have 100 fw rules each marking packet as 1 to 100. I am
marking these to do traffic shaping, so that I need not check fw
matching conditions on every packet. Simple check on mark will be
sufficient.
iptables -t mangle  -A Forward -j mark --restore-mark
iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept

iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
condition 1> -j MARK --set-mark 1

iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
condition 2> -j MARK --set-mark 2
.
.
.
iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
condition 100> -j MARK --set-mark 100

iptables -t mangle -A Forward -j Connmark --save-mark

Next would be Filter table in Forward chain:

iptables -t filter -m connmark ! --mark 0 -j Accept

Note that as we are using connmark so we don't require related,
established rule.

Now as I have to do bw shaping, so I need 100 tc filter rules, something like

tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1

tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 2 fw flowid 1:2

tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 3 fw flowid 1:3
.
.
.
tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
(pardon me if I am wrong on syntax, idea is to give a feel of things)

Now visualize traffic for rule 100. every pkt, in tc, will face a
delay equal 100T where T is the time to search first entry, as search
will be linear. Clearly this doesn't scale well when rule count moves
to thousand or more.

However, if we have an ipset hash:mark with skbinfo support, then we
can store this mark - tc_class membership in it and then with a
constant lookup time we can scale to any no. of rules with cost being
only memory:

ipset -N mark_tc_class_map hash:mark skbinfo

ipset -A mark_tc_class_map 1 skbprio 1:1
ipset -A mark_tc_class_map 2 skbprio 1:2
ipset -A mark_tc_class_map 3 skbprio 1:3
.
.
.
ipset -A mark_tc_class_map 4 skbprio 1:100

Please note that this is not storing mark in skbinfo but creating hash
of marks and then storing skbinfo against each mark.

This ipset then we will use in mangle chain of postrouting

iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
With above rule we don't require those 100 tc filters mentioned above.
It all reduces to single rule in iptables and constant lookup time for
traffic shaping.



On 9/7/15, Jozsef Kadlecsik <kadlec@blackhole.kfki.hu> wrote:
> On Sun, 6 Sep 2015, Akshat Kakkar wrote:
>
>> With latest addition of storing skbinfo (mainly skbprio) in ipset and
>> then applying it later to the traffic as and when it passes the iptables
>> ruleset, it becomes relatively easy and simple to do traffic shaping.
>>
>> IMHO what one of the feature set which we can add is to have an ipset of
>> only fwmarks i.e. fw-marks will be hashed and stored in the ipset and
>> then later using these marks we can instantly (in single look up) set
>> traffic class.
>
> You can already store skbmark in the set element extension and set the
> mark value to the matching packets.
>
>> Motivation for this is the fact that just reading the mark can signify
>> which fw rule it belongs to (no matter how complex was that firewall
>> rule). So if we do traffic shaping based on marks, it becomes
>> straightforward traffic shaping for that fw rule. Now as no of rules
>> increase, this lookup will also increase linearly and
>> more-importantly, we cant take advantage of any statefulness over
>> here, i.e. if there are 100 rules then everytime my packet will have
>> to go down 100 lines, get it matched, get its skbprio value and then
>> move forward.
>
> Sorry, but I don't undestand this...
>
>> however, if we such an ipset as I am mentioning, so all these skbprio
>> settings and all can always be done in single shot,
>>
>> say we make something like
>> ipset -N MARK-TC-MAP hash:mark skbinfo
>>
>> iptables -t mangle -A POSTROUTING -j SET --map-set MARK-TC-MAP src
>> --map-prio
>
> ...and your example doesn't help either. Why do you need the set at all?
> You could simply write
>
> iptables -t mangle -A POSTROUTING -j MARK --set-mark value
>
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary
>

Re: Feature suggestion ...

[Jozsef Kadlecsik]

On Mon, 7 Sep 2015, Akshat Kakkar wrote:

> I am suggesting an ipset hash:mark.
> 
> Let me explain the motivation for this requirement:
> 
> Assume we have 100 fw rules each marking packet as 1 to 100. I am
> marking these to do traffic shaping, so that I need not check fw
> matching conditions on every packet. Simple check on mark will be
> sufficient.
> iptables -t mangle  -A Forward -j mark --restore-mark
> iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept
> 
> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
> condition 1> -j MARK --set-mark 1
> .
> .
> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
> condition 100> -j MARK --set-mark 100
> 
> iptables -t mangle -A Forward -j Connmark --save-mark
> 
> Next would be Filter table in Forward chain:
> 
> iptables -t filter -m connmark ! --mark 0 -j Accept
> 
> Note that as we are using connmark so we don't require related,
> established rule.
> 
> Now as I have to do bw shaping, so I need 100 tc filter rules, something like
> 
> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1
> .
> .
> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
> (pardon me if I am wrong on syntax, idea is to give a feel of things)
> 
> Now visualize traffic for rule 100. every pkt, in tc, will face a
> delay equal 100T where T is the time to search first entry, as search
> will be linear. Clearly this doesn't scale well when rule count moves
> to thousand or more.

tc is not bad at evaluating large number of rules. You should compare 
measured performances instead of assuming those.

> However, if we have an ipset hash:mark with skbinfo support, then we
> can store this mark - tc_class membership in it and then with a
> constant lookup time we can scale to any no. of rules with cost being
> only memory:
> 
> ipset -N mark_tc_class_map hash:mark skbinfo
> 
> ipset -A mark_tc_class_map 1 skbprio 1:1
> .
> ipset -A mark_tc_class_map 4 skbprio 1:100
> 
> Please note that this is not storing mark in skbinfo but creating hash
> of marks and then storing skbinfo against each mark.
> 
> This ipset then we will use in mangle chain of postrouting
> 
> iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
> With above rule we don't require those 100 tc filters mentioned above.
> It all reduces to single rule in iptables and constant lookup time for
> traffic shaping.

You can already do this with the hash:ip,mark type if your rules allow 
reducing the conditions to IP address + mark value pairs.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: Feature suggestion ...

[Akshat Kakkar]

On Mon, Sep 7, 2015 at 1:53 AM, Jozsef Kadlecsik
<kadlec@blackhole.kfki.hu> wrote:
> On Mon, 7 Sep 2015, Akshat Kakkar wrote:
>
>> I am suggesting an ipset hash:mark.
>>
>> Let me explain the motivation for this requirement:
>>
>> Assume we have 100 fw rules each marking packet as 1 to 100. I am
>> marking these to do traffic shaping, so that I need not check fw
>> matching conditions on every packet. Simple check on mark will be
>> sufficient.
>> iptables -t mangle  -A Forward -j mark --restore-mark
>> iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept
>>
>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>> condition 1> -j MARK --set-mark 1
>> .
>> .
>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>> condition 100> -j MARK --set-mark 100
>>
>> iptables -t mangle -A Forward -j Connmark --save-mark
>>
>> Next would be Filter table in Forward chain:
>>
>> iptables -t filter -m connmark ! --mark 0 -j Accept
>>
>> Note that as we are using connmark so we don't require related,
>> established rule.
>>
>> Now as I have to do bw shaping, so I need 100 tc filter rules, something like
>>
>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1
>> .
>> .
>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
>> (pardon me if I am wrong on syntax, idea is to give a feel of things)
>>
>> Now visualize traffic for rule 100. every pkt, in tc, will face a
>> delay equal 100T where T is the time to search first entry, as search
>> will be linear. Clearly this doesn't scale well when rule count moves
>> to thousand or more.
>
> tc is not bad at evaluating large number of rules. You should compare
> measured performances instead of assuming those.
>
>> However, if we have an ipset hash:mark with skbinfo support, then we
>> can store this mark - tc_class membership in it and then with a
>> constant lookup time we can scale to any no. of rules with cost being
>> only memory:
>>
>> ipset -N mark_tc_class_map hash:mark skbinfo
>>
>> ipset -A mark_tc_class_map 1 skbprio 1:1
>> .
>> ipset -A mark_tc_class_map 4 skbprio 1:100
>>
>> Please note that this is not storing mark in skbinfo but creating hash
>> of marks and then storing skbinfo against each mark.
>>
>> This ipset then we will use in mangle chain of postrouting
>>
>> iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
>> With above rule we don't require those 100 tc filters mentioned above.
>> It all reduces to single rule in iptables and constant lookup time for
>> traffic shaping.
>
> You can already do this with the hash:ip,mark type if your rules allow
> reducing the conditions to IP address + mark value pairs.

Well, I am having some mix of rules. Some are per IP bandwidth shaping
rules. So that I have taken care by hash:ip,mark.
However, there are other rules also, same as the one I have mentioned
above in example. So if I use tc filter for these rules, then my per
IP bandwidth limited traffic unnecessarily has to pass through all
those filters, which in the presence of ipset:mark will also go to tc
class directly.

Re: Feature suggestion ...

[Anton Danilov]

Hello.
You can use the firewall mark as tc classid with fwmark filter.
It can interpret fwmark as tc classid. So, you can use this feature
inside your ipset/iptables/tc rules.

2015-09-07 15:09 GMT+03:00 Akshat Kakkar <akshat.1984@gmail.com>:
> On Mon, Sep 7, 2015 at 1:53 AM, Jozsef Kadlecsik
> <kadlec@blackhole.kfki.hu> wrote:
>> On Mon, 7 Sep 2015, Akshat Kakkar wrote:
>>
>>> I am suggesting an ipset hash:mark.
>>>
>>> Let me explain the motivation for this requirement:
>>>
>>> Assume we have 100 fw rules each marking packet as 1 to 100. I am
>>> marking these to do traffic shaping, so that I need not check fw
>>> matching conditions on every packet. Simple check on mark will be
>>> sufficient.
>>> iptables -t mangle  -A Forward -j mark --restore-mark
>>> iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept
>>>
>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>> condition 1> -j MARK --set-mark 1
>>> .
>>> .
>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>> condition 100> -j MARK --set-mark 100
>>>
>>> iptables -t mangle -A Forward -j Connmark --save-mark
>>>
>>> Next would be Filter table in Forward chain:
>>>
>>> iptables -t filter -m connmark ! --mark 0 -j Accept
>>>
>>> Note that as we are using connmark so we don't require related,
>>> established rule.
>>>
>>> Now as I have to do bw shaping, so I need 100 tc filter rules, something like
>>>
>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1
>>> .
>>> .
>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
>>> (pardon me if I am wrong on syntax, idea is to give a feel of things)
>>>
>>> Now visualize traffic for rule 100. every pkt, in tc, will face a
>>> delay equal 100T where T is the time to search first entry, as search
>>> will be linear. Clearly this doesn't scale well when rule count moves
>>> to thousand or more.
>>
>> tc is not bad at evaluating large number of rules. You should compare
>> measured performances instead of assuming those.
>>
>>> However, if we have an ipset hash:mark with skbinfo support, then we
>>> can store this mark - tc_class membership in it and then with a
>>> constant lookup time we can scale to any no. of rules with cost being
>>> only memory:
>>>
>>> ipset -N mark_tc_class_map hash:mark skbinfo
>>>
>>> ipset -A mark_tc_class_map 1 skbprio 1:1
>>> .
>>> ipset -A mark_tc_class_map 4 skbprio 1:100
>>>
>>> Please note that this is not storing mark in skbinfo but creating hash
>>> of marks and then storing skbinfo against each mark.
>>>
>>> This ipset then we will use in mangle chain of postrouting
>>>
>>> iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
>>> With above rule we don't require those 100 tc filters mentioned above.
>>> It all reduces to single rule in iptables and constant lookup time for
>>> traffic shaping.
>>
>> You can already do this with the hash:ip,mark type if your rules allow
>> reducing the conditions to IP address + mark value pairs.
>
> Well, I am having some mix of rules. Some are per IP bandwidth shaping
> rules. So that I have taken care by hash:ip,mark.
> However, there are other rules also, same as the one I have mentioned
> above in example. So if I use tc filter for these rules, then my per
> IP bandwidth limited traffic unnecessarily has to pass through all
> those filters, which in the presence of ipset:mark will also go to tc
> class directly.
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Anton.

Re: Feature suggestion ...

[Akshat Kakkar]

hi!
I am sorry to say but I am unable to comprehend that. Can you give an example?

On Mon, Sep 7, 2015 at 6:22 PM, Anton Danilov
<littlesmilingcloud@gmail.com> wrote:
> Hello.
> You can use the firewall mark as tc classid with fwmark filter.
> It can interpret fwmark as tc classid. So, you can use this feature
> inside your ipset/iptables/tc rules.
>
> 2015-09-07 15:09 GMT+03:00 Akshat Kakkar <akshat.1984@gmail.com>:
>> On Mon, Sep 7, 2015 at 1:53 AM, Jozsef Kadlecsik
>> <kadlec@blackhole.kfki.hu> wrote:
>>> On Mon, 7 Sep 2015, Akshat Kakkar wrote:
>>>
>>>> I am suggesting an ipset hash:mark.
>>>>
>>>> Let me explain the motivation for this requirement:
>>>>
>>>> Assume we have 100 fw rules each marking packet as 1 to 100. I am
>>>> marking these to do traffic shaping, so that I need not check fw
>>>> matching conditions on every packet. Simple check on mark will be
>>>> sufficient.
>>>> iptables -t mangle  -A Forward -j mark --restore-mark
>>>> iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept
>>>>
>>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>>> condition 1> -j MARK --set-mark 1
>>>> .
>>>> .
>>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>>> condition 100> -j MARK --set-mark 100
>>>>
>>>> iptables -t mangle -A Forward -j Connmark --save-mark
>>>>
>>>> Next would be Filter table in Forward chain:
>>>>
>>>> iptables -t filter -m connmark ! --mark 0 -j Accept
>>>>
>>>> Note that as we are using connmark so we don't require related,
>>>> established rule.
>>>>
>>>> Now as I have to do bw shaping, so I need 100 tc filter rules, something like
>>>>
>>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1
>>>> .
>>>> .
>>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
>>>> (pardon me if I am wrong on syntax, idea is to give a feel of things)
>>>>
>>>> Now visualize traffic for rule 100. every pkt, in tc, will face a
>>>> delay equal 100T where T is the time to search first entry, as search
>>>> will be linear. Clearly this doesn't scale well when rule count moves
>>>> to thousand or more.
>>>
>>> tc is not bad at evaluating large number of rules. You should compare
>>> measured performances instead of assuming those.
>>>
>>>> However, if we have an ipset hash:mark with skbinfo support, then we
>>>> can store this mark - tc_class membership in it and then with a
>>>> constant lookup time we can scale to any no. of rules with cost being
>>>> only memory:
>>>>
>>>> ipset -N mark_tc_class_map hash:mark skbinfo
>>>>
>>>> ipset -A mark_tc_class_map 1 skbprio 1:1
>>>> .
>>>> ipset -A mark_tc_class_map 4 skbprio 1:100
>>>>
>>>> Please note that this is not storing mark in skbinfo but creating hash
>>>> of marks and then storing skbinfo against each mark.
>>>>
>>>> This ipset then we will use in mangle chain of postrouting
>>>>
>>>> iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
>>>> With above rule we don't require those 100 tc filters mentioned above.
>>>> It all reduces to single rule in iptables and constant lookup time for
>>>> traffic shaping.
>>>
>>> You can already do this with the hash:ip,mark type if your rules allow
>>> reducing the conditions to IP address + mark value pairs.
>>
>> Well, I am having some mix of rules. Some are per IP bandwidth shaping
>> rules. So that I have taken care by hash:ip,mark.
>> However, there are other rules also, same as the one I have mentioned
>> above in example. So if I use tc filter for these rules, then my per
>> IP bandwidth limited traffic unnecessarily has to pass through all
>> those filters, which in the presence of ipset:mark will also go to tc
>> class directly.
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Anton.

Re: Feature suggestion ...

[Akshat Kakkar]

Hi Anton, Jozsef!

If comfortable, then please reply!

-Akshat

On Mon, Sep 7, 2015 at 6:26 PM, Akshat Kakkar <akshat.1984@gmail.com> wrote:
> hi!
> I am sorry to say but I am unable to comprehend that. Can you give an example?
>
> On Mon, Sep 7, 2015 at 6:22 PM, Anton Danilov
> <littlesmilingcloud@gmail.com> wrote:
>> Hello.
>> You can use the firewall mark as tc classid with fwmark filter.
>> It can interpret fwmark as tc classid. So, you can use this feature
>> inside your ipset/iptables/tc rules.
>>
>> 2015-09-07 15:09 GMT+03:00 Akshat Kakkar <akshat.1984@gmail.com>:
>>> On Mon, Sep 7, 2015 at 1:53 AM, Jozsef Kadlecsik
>>> <kadlec@blackhole.kfki.hu> wrote:
>>>> On Mon, 7 Sep 2015, Akshat Kakkar wrote:
>>>>
>>>>> I am suggesting an ipset hash:mark.
>>>>>
>>>>> Let me explain the motivation for this requirement:
>>>>>
>>>>> Assume we have 100 fw rules each marking packet as 1 to 100. I am
>>>>> marking these to do traffic shaping, so that I need not check fw
>>>>> matching conditions on every packet. Simple check on mark will be
>>>>> sufficient.
>>>>> iptables -t mangle  -A Forward -j mark --restore-mark
>>>>> iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept
>>>>>
>>>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>>>> condition 1> -j MARK --set-mark 1
>>>>> .
>>>>> .
>>>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>>>> condition 100> -j MARK --set-mark 100
>>>>>
>>>>> iptables -t mangle -A Forward -j Connmark --save-mark
>>>>>
>>>>> Next would be Filter table in Forward chain:
>>>>>
>>>>> iptables -t filter -m connmark ! --mark 0 -j Accept
>>>>>
>>>>> Note that as we are using connmark so we don't require related,
>>>>> established rule.
>>>>>
>>>>> Now as I have to do bw shaping, so I need 100 tc filter rules, something like
>>>>>
>>>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1
>>>>> .
>>>>> .
>>>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
>>>>> (pardon me if I am wrong on syntax, idea is to give a feel of things)
>>>>>
>>>>> Now visualize traffic for rule 100. every pkt, in tc, will face a
>>>>> delay equal 100T where T is the time to search first entry, as search
>>>>> will be linear. Clearly this doesn't scale well when rule count moves
>>>>> to thousand or more.
>>>>
>>>> tc is not bad at evaluating large number of rules. You should compare
>>>> measured performances instead of assuming those.
>>>>
>>>>> However, if we have an ipset hash:mark with skbinfo support, then we
>>>>> can store this mark - tc_class membership in it and then with a
>>>>> constant lookup time we can scale to any no. of rules with cost being
>>>>> only memory:
>>>>>
>>>>> ipset -N mark_tc_class_map hash:mark skbinfo
>>>>>
>>>>> ipset -A mark_tc_class_map 1 skbprio 1:1
>>>>> .
>>>>> ipset -A mark_tc_class_map 4 skbprio 1:100
>>>>>
>>>>> Please note that this is not storing mark in skbinfo but creating hash
>>>>> of marks and then storing skbinfo against each mark.
>>>>>
>>>>> This ipset then we will use in mangle chain of postrouting
>>>>>
>>>>> iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
>>>>> With above rule we don't require those 100 tc filters mentioned above.
>>>>> It all reduces to single rule in iptables and constant lookup time for
>>>>> traffic shaping.
>>>>
>>>> You can already do this with the hash:ip,mark type if your rules allow
>>>> reducing the conditions to IP address + mark value pairs.
>>>
>>> Well, I am having some mix of rules. Some are per IP bandwidth shaping
>>> rules. So that I have taken care by hash:ip,mark.
>>> However, there are other rules also, same as the one I have mentioned
>>> above in example. So if I use tc filter for these rules, then my per
>>> IP bandwidth limited traffic unnecessarily has to pass through all
>>> those filters, which in the presence of ipset:mark will also go to tc
>>> class directly.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>> --
>> Anton.

Re: Feature suggestion ...

[Pablo Neira Ayuso]

On Tue, Sep 08, 2015 at 10:45:41AM +0530, Akshat Kakkar wrote:
> Hi Anton, Jozsef!
> 
> If comfortable, then please reply!

Please, don't top-post:

http://www.netfilter.org/mailinglists.html#list-rules

Thanks.

Re: Feature suggestion ...

[Akshat Kakkar]

Hi Jozsef,

Well you are right. Tc is not bad at evaluating large no. of rules but
only if it is only fwmark rules.
if it is using u32 filters, then performace do degrade severely.

But this raise another doubt.
Why is that so? Does simple fwmark filters are stored in hash? I am
asking this because when I list the filters, there is no order in the
list.
So, are fwmark filters hashed?

On Mon, Sep 7, 2015 at 5:39 PM, Akshat Kakkar <akshat.1984@gmail.com> wrote:
> On Mon, Sep 7, 2015 at 1:53 AM, Jozsef Kadlecsik
> <kadlec@blackhole.kfki.hu> wrote:
>> On Mon, 7 Sep 2015, Akshat Kakkar wrote:
>>
>>> I am suggesting an ipset hash:mark.
>>>
>>> Let me explain the motivation for this requirement:
>>>
>>> Assume we have 100 fw rules each marking packet as 1 to 100. I am
>>> marking these to do traffic shaping, so that I need not check fw
>>> matching conditions on every packet. Simple check on mark will be
>>> sufficient.
>>> iptables -t mangle  -A Forward -j mark --restore-mark
>>> iptables -t mangle -A Forward -m mark ! --mark 0 -j Accept
>>>
>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>> condition 1> -j MARK --set-mark 1
>>> .
>>> .
>>> iptables -t mangle -A FORWARD -i etho -o eth1 <firewall match
>>> condition 100> -j MARK --set-mark 100
>>>
>>> iptables -t mangle -A Forward -j Connmark --save-mark
>>>
>>> Next would be Filter table in Forward chain:
>>>
>>> iptables -t filter -m connmark ! --mark 0 -j Accept
>>>
>>> Note that as we are using connmark so we don't require related,
>>> established rule.
>>>
>>> Now as I have to do bw shaping, so I need 100 tc filter rules, something like
>>>
>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:1
>>> .
>>> .
>>> tc filter add dev eth1 protocol ip parent 1:0 prio 1 handle 100 fw flowid 1:100
>>> (pardon me if I am wrong on syntax, idea is to give a feel of things)
>>>
>>> Now visualize traffic for rule 100. every pkt, in tc, will face a
>>> delay equal 100T where T is the time to search first entry, as search
>>> will be linear. Clearly this doesn't scale well when rule count moves
>>> to thousand or more.
>>
>> tc is not bad at evaluating large number of rules. You should compare
>> measured performances instead of assuming those.
>>
>>> However, if we have an ipset hash:mark with skbinfo support, then we
>>> can store this mark - tc_class membership in it and then with a
>>> constant lookup time we can scale to any no. of rules with cost being
>>> only memory:
>>>
>>> ipset -N mark_tc_class_map hash:mark skbinfo
>>>
>>> ipset -A mark_tc_class_map 1 skbprio 1:1
>>> .
>>> ipset -A mark_tc_class_map 4 skbprio 1:100
>>>
>>> Please note that this is not storing mark in skbinfo but creating hash
>>> of marks and then storing skbinfo against each mark.
>>>
>>> This ipset then we will use in mangle chain of postrouting
>>>
>>> iptables -t mangle -A Postrouting  -j Set --map-set mark_tc_class_map --map-prio
>>> With above rule we don't require those 100 tc filters mentioned above.
>>> It all reduces to single rule in iptables and constant lookup time for
>>> traffic shaping.
>>
>> You can already do this with the hash:ip,mark type if your rules allow
>> reducing the conditions to IP address + mark value pairs.
>
> Well, I am having some mix of rules. Some are per IP bandwidth shaping
> rules. So that I have taken care by hash:ip,mark.
> However, there are other rules also, same as the one I have mentioned
> above in example. So if I use tc filter for these rules, then my per
> IP bandwidth limited traffic unnecessarily has to pass through all
> those filters, which in the presence of ipset:mark will also go to tc
> class directly.

Re: Feature suggestion ...

[Akshat Kakkar]

On Mon, Sep 7, 2015 at 6:22 PM, Anton Danilov
<littlesmilingcloud@gmail.com> wrote:
> Hello.
> You can use the firewall mark as tc classid with fwmark filter.
> It can interpret fwmark as tc classid. So, you can use this feature
> inside your ipset/iptables/tc rules.
>

Can you provide an example as how fwmark is intrepreted as classid ?

Re: Feature suggestion ...

[Akshat Kakkar]

That's gr8.

I think we should document this behaviour. It is very useful.

On Wed, Sep 9, 2015 at 9:35 PM, Anton Danilov
<littlesmilingcloud@gmail.com> wrote:
> Without any options fw classifier maps fwmark to classid.
>
> tc filter add dev <iface> parent <qhandle> protocol ip prio 1 fw
>
> 2015-09-09 9:35 GMT+03:00 Akshat Kakkar <akshat.1984@gmail.com>:
>> On Mon, Sep 7, 2015 at 6:22 PM, Anton Danilov
>> <littlesmilingcloud@gmail.com> wrote:
>>> Hello.
>>> You can use the firewall mark as tc classid with fwmark filter.
>>> It can interpret fwmark as tc classid. So, you can use this feature
>>> inside your ipset/iptables/tc rules.
>>>
>>
>> Can you provide an example as how fwmark is intrepreted as classid ?
>
>
>
> --
> Anton.

Re: Feature suggestion ...

[Akshat Kakkar]

>> Without any options fw classifier maps fwmark to classid.
>>
>> tc filter add dev <iface> parent <qhandle> protocol ip prio 1 fw

What if the packet has fwmark and skbprio set?

Re: Feature suggestion ...

[Anton Danilov]

Classid will be overwritten by fw classifier.

2015-09-10 13:20 GMT+03:00 Akshat Kakkar <akshat.1984@gmail.com>:
>>> Without any options fw classifier maps fwmark to classid.
>>>
>>> tc filter add dev <iface> parent <qhandle> protocol ip prio 1 fw
>
> What if the packet has fwmark and skbprio set?



-- 
Anton.

Re: Feature suggestion ...

[Akshat Kakkar]

> Classid will be overwritten by fw classifier.
>
For sake of completeness,

So if I use tc filter add dev <iface> parent <qhandle> protocol ip prio 1 fw

and my packet has mark(640001) and classid(1:10) both set,
then because of above tc filter, the exisiting classid will be set to
64:1 i.e. equal to the mark.

Is my understanding correct?

If my understanding is correct,
shouldn't it be like that if class is set, then it should not overwrite?



> 2015-09-10 13:20 GMT+03:00 Akshat Kakkar <akshat.1984@gmail.com>:
>>>> Without any options fw classifier maps fwmark to classid.
>>>>
>>>> tc filter add dev <iface> parent <qhandle> protocol ip prio 1 fw
>>
>> What if the packet has fwmark and skbprio set?