[ANNOUNCE] ipset 6.11 released

[ANNOUNCE] ipset 6.11 released

[Jozsef Kadlecsik]

Hi,

I'm happy to announce ipset 6.11, which brings a couple of build fixes 
from Jan and a timeout bugfix for the hash:net,iface type where the timed 
out entries were still matched until the garbage collector erased them.

The new version includes two new features as well:

- The hash:*net* types support the "nomatch" flag for the entries,
  by which one can build up a set with exceptions. For example the set

	ipset create test hash:net
	ipset add test 192.168.0/24
	ipset add test 192.168.0/30 nomatch

   matches all IP addresses from 192.168.0/24 except the ones
   from 192.168.0/30. 

- Escaping notation added to support hostnames and service names with dash 
  in the name: such names must be enclosed by square brackets

	ipset add test [hostname-with-dash],[ftp-data]

Userspace changes:
 - Support hostnames and service names with dash
 - Exceptions support added to hash:*net* types
 - Set types moved into libipset library
 - Library map file added in order to support library versioning
 - doc: Linux 2.6.39 already has the defs (Jan Engelhardt)
 - build: install libipset in the right place (Jan Engelhardt)
 - Provide a pkgconfig file (Jan Engelhardt)
 - build: make distcheck work and use POSIX mode for tarball generation
   (Jan Engelhardt)
 - build: install libipset/linux_ip_set_list.h (Jan Engelhardt)
 - build: include libipset/nfproto.h (Jan Engelhardt)
 - build: process include/libipset/ (Jan Engelhardt)
 - build: use AC_CONFIG_AUX_DIR and stash away tools (Jan Engelhardt)
 - Update .gitignore (Jan Engelhardt)

Kernel part changes:
 - hash:net,iface timeout bug fixed
 - Exceptions support added to hash:*net* types
 - net: Add export.h for EXPORT_SYMBOL/THIS_MODULE to non-modules
 - Log warning when a hash type of set gets full

You can download the source code of ipset from:
        http://ipset.netfilter.org
        ftp://ftp.netfilter.org/pub/ipset/
        git://git.netfilter.org/ipset.git

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.11 released

[Mr Dash Four]

> Kernel part changes:
>  - hash:net,iface timeout bug fixed
>  - Exceptions support added to hash:*net* types
>   
Any chance of fixing this bug soon:

~# ipset n test hash:net family inet timeout 0
~# ipset a test 10.1.0.0/16
~# ipset t test 10.1.12.12
10.1.12.12 is in set test.
~# ipset t test 10.1.12.0/24
10.1.12.0/24 is NOT in test.

Re: [ANNOUNCE] ipset 6.11 released

[Jozsef Kadlecsik]

On Sun, 15 Jan 2012, Mr Dash Four wrote:

> > Kernel part changes:
> >  - hash:net,iface timeout bug fixed
> >  - Exceptions support added to hash:*net* types
> >   
> Any chance of fixing this bug soon:
> 
> ~# ipset n test hash:net family inet timeout 0
> ~# ipset a test 10.1.0.0/16
> ~# ipset t test 10.1.12.12
> 10.1.12.12 is in set test.
> ~# ipset t test 10.1.12.0/24
> 10.1.12.0/24 is NOT in test.

It's a feature which I'm not going to fix in any near future.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.11 released

[Mr Dash Four]

>> Any chance of fixing this bug soon:
>>
>> ~# ipset n test hash:net family inet timeout 0
>> ~# ipset a test 10.1.0.0/16
>> ~# ipset t test 10.1.12.12
>> 10.1.12.12 is in set test.
>> ~# ipset t test 10.1.12.0/24
>> 10.1.12.0/24 is NOT in test.
>>     
>
> It's a feature which I'm not going to fix in any near future.
>   
It isn't a "feature", it is a bug: 10.1.12.0/24 is within the 
10.1.0.0/16 range, so the above test should return true, not false. 
Either that, or ip range values should be restricted/excluded from the 
"test" command in the ipset userspace binary.

Re: [ANNOUNCE] ipset 6.11 released

[Jozsef Kadlecsik]

On Sun, 15 Jan 2012, Mr Dash Four wrote:

> > > Any chance of fixing this bug soon:
> > > 
> > > ~# ipset n test hash:net family inet timeout 0
> > > ~# ipset a test 10.1.0.0/16
> > > ~# ipset t test 10.1.12.12
> > > 10.1.12.12 is in set test.
> > > ~# ipset t test 10.1.12.0/24
> > > 10.1.12.0/24 is NOT in test.
> > 
> > It's a feature which I'm not going to fix in any near future.
> >   
> It isn't a "feature", it is a bug: 10.1.12.0/24 is within the 10.1.0.0/16
> range, so the above test should return true, not false. Either that, or ip
> range values should be restricted/excluded from the "test" command in the
> ipset userspace binary.

The "test" functionality is already overloaded. It has two "modes":

- you can test how the *kernel* sees the set, when checking a single IP
  address
- you can check whether an *exact* element is added to the set or not.

As the first one overloads the second one, for hash:*net* types the second 
mode is already "incomplete" in the sense that one cannot check whether a 
given single IP address is already added to a hash:*net* type of set as an 
exact element or not, because a network element may match it.

Your request means a third mode, which could lead to even more confusion, 
because that way one could not check whether the tested address as 
*element* is added to the set or not.

There's no magical element-aggregation in the hash:* types. That is, even 
if 10.1.0.0/16 is added as an element, 10.1.0.0/24 can be added again as 
an independent element: either it should be rejected (when the command was 
issued without the --exist flag) or silently ignored (when was issued with 
it). So even to consider your feature requests, it could come only after 
implementing element-aggregation.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.11 released

[Mr Dash Four]

> Your request means a third mode, which could lead to even more confusion, 
> because that way one could not check whether the tested address as 
> *element* is added to the set or not.
>   
This is not a feature request, it is a bug fix!

If ipset, for whatever reason, can't (or won't) handle ip range for 
match testing, then such requests should be discarded with the 
appropriate error or warning message on the console given by the ipset 
binary, or, the appropriate functionality should be implemented so that 
such test matches give the right result.

Neither of this currently happens - ipset accepts the ip range value and 
then outputs the wrong result (indicating that there is no match where 
it clearly is). So, what I have highlighted in my previous posts is, 
evidently, *not* a feature request, but a *bug fix*.

> There's no magical element-aggregation in the hash:* types. That is, even 
> if 10.1.0.0/16 is added as an element, 10.1.0.0/24 can be added again as 
> an independent element: either it should be rejected (when the command was 
> issued without the --exist flag) or silently ignored (when was issued with 
> it).
I am not very familiar with the inner intricacies of ipset (I am just a 
user at the end of the day), so I can't comment on whether this ip range 
match could (or should) be implemented, but the way ipset now works is 
wrong, unless you believe that the 10.1.12.0/24 ip range is not matched, 
in its entirety, by the 10.1.0.0/16 one. Not to mention that the above 
bug is a clear *regression* as the v4.x version of the ipset binary was 
able, for whatever reason , to produce matches - successfully - using ip 
ranges like the one I used as an example in my initial post.

> So even to consider your feature requests, it could come only after 
> implementing element-aggregation.
>   
Again, this is *not* a feature request - it is a bug fix and I think I 
already pointed out above why that is so.

Re: [ANNOUNCE] ipset 6.11 released

[Jozsef Kadlecsik]

On Sun, 15 Jan 2012, Mr Dash Four wrote:

> > Your request means a third mode, which could lead to even more confusion,
> > because that way one could not check whether the tested address as
> > *element* is added to the set or not.
> >   
> This is not a feature request, it is a bug fix!
> 
> If ipset, for whatever reason, can't (or won't) handle ip range for match
> testing, then such requests should be discarded with the appropriate error or
> warning message on the console given by the ipset binary, or, the appropriate
> functionality should be implemented so that such test matches give the right
> result.

ipset is a tool to build up so called sets inside the Linux kernel. The 
sets have any use in the kernel side only and there the kernel matches 
single IP addresses and never whole networks.

But if you say so, let it be a bug. Won't be fixed in the near future, 
however, as it has got a very low priority.

> Not to mention that the above bug is a clear *regression* as the v4.x 
> version of the ipset binary was able, for whatever reason , to produce 
> matches - successfully - using ip ranges like the one I used as an 
> example in my initial post.

The single set type in v4.x which incidentally had the feature you are 
missing was the iptreemap type alone. The *hash set types in ipset 4.x 
behaved exactly the same way as the current ones (ignoring the fact that 
the 4.x versions had more limitations and less functionality than the 
successor hash:* types in ipset 6.x).

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.11 released

[Mr Dash Four]

> ipset is a tool to build up so called sets inside the Linux kernel.
I think I know what ipset is, thank you.

>  The 
> sets have any use in the kernel side only and there the kernel matches 
> single IP addresses and never whole networks.
>   
OK, I don't have intimate knowledge of the ipset code and its internal 
workings, but it obviously accepts IP ranges since if I have a hash:net 
set containing 10.1.0.0/16 for example and then test for that exact IP 
range (10.1.0.0/16) then the test returns true, so ipset obviously 
processes this IP range and returns a good result. How is that done if 
the kernel "matches single IP addresses and never whole networks" then?

One other thing: *if* ipset can only accept single IP addresses instead 
of IP ranges (I don't believe this to be the case, but anyway, if it 
does), then you could process a single IP address in a loop containing 
the whole range to be tested (10.1.12.0/24 in my example - i.e. looping 
from 10.1.12.0 until 10.1.12.255 inclusive) and bail out as soon as 
there is no match, which would then return 'false' (i.e. no match). You 
could even speed things up a bit by implementing batch processing of IP 
ranges internally (via a single kernel APIs instead of looping via ipset 
and calling the kernel API each time for a single IP address check).

I know this implementation is a bit crude, but since this testing takes 
place in userspace then this delay won't matter *that* much. How doable 
is that?

Re: [ANNOUNCE] ipset 6.11 released

[Jozsef Kadlecsik]

On Wed, 18 Jan 2012, Mr Dash Four wrote:

> >  The sets have any use in the kernel side only and there the kernel 
> > matches single IP addresses and never whole networks.
> >   
> OK, I don't have intimate knowledge of the ipset code and its internal 
> workings, but it obviously accepts IP ranges since if I have a hash:net 
> set containing 10.1.0.0/16 for example and then test for that exact IP 
> range (10.1.0.0/16) then the test returns true, so ipset obviously 
> processes this IP range and returns a good result. How is that done if 
> the kernel "matches single IP addresses and never whole networks" then?

The kernel uses the sets to match the source/destination addresses of IP 
packets. And those are host, that is single IP addresses (not counting 
broadcast, multicast addresses).
 
> One other thing: *if* ipset can only accept single IP addresses instead 
> of IP ranges (I don't believe this to be the case, but anyway, if it 
> does), then you could process a single IP address in a loop containing 
> the whole range to be tested (10.1.12.0/24 in my example - i.e. looping 
> from 10.1.12.0 until 10.1.12.255 inclusive) and bail out as soon as 
> there is no match, which would then return 'false' (i.e. no match). You 
> could even speed things up a bit by implementing batch processing of IP 
> ranges internally (via a single kernel APIs instead of looping via ipset 
> and calling the kernel API each time for a single IP address check).
> 
> I know this implementation is a bit crude, but since this testing takes 
> place in userspace then this delay won't matter *that* much. How doable 
> is that?

The userspace testing is passed to the kernel for execution. There's no
shadow or backup or whatever set in userspace. The sets exist in kernel 
space and therefore all operations happen there.

Why do you need such tests at all?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.11 released

[Mr Dash Four]

>> One other thing: *if* ipset can only accept single IP addresses instead 
>> of IP ranges (I don't believe this to be the case, but anyway, if it 
>> does), then you could process a single IP address in a loop containing 
>> the whole range to be tested (10.1.12.0/24 in my example - i.e. looping 
>> from 10.1.12.0 until 10.1.12.255 inclusive) and bail out as soon as 
>> there is no match, which would then return 'false' (i.e. no match). You 
>> could even speed things up a bit by implementing batch processing of IP 
>> ranges internally (via a single kernel APIs instead of looping via ipset 
>> and calling the kernel API each time for a single IP address check).
>>
>> I know this implementation is a bit crude, but since this testing takes 
>> place in userspace then this delay won't matter *that* much. How doable 
>> is that?
>>     
>
> The userspace testing is passed to the kernel for execution. There's no
> shadow or backup or whatever set in userspace. The sets exist in kernel 
> space and therefore all operations happen there.
>   
So, is the above doable in any shape or form or not?

> Why do you need such tests at all?
>   
Various reasons. Two common uses (at least in my case) would be to test 
ip range against quite a large set of registered subnets (taken from the 
geoip database and sorted using my own criteria). The tested ip range is 
either candidate to 1). ban that network, in which case I do not want 
duplicates if the existing range is already there; or 2) include the ip 
range in a separate set, making sure that it is not already in the 
'banned' set and also that it is not already included in the 'customer' 
set (which although limited, has quite significant number of set members 
- usually small subnets).

Because the ip range in ipset is not working correctly, I had to first 
manually go over the testing ranges. Later on I devised a small shell 
script doing what I just described in my previous post (quoted above), 
but it is quite ugly and inefficient - it would be much better if ip 
range testing in ipset was functioning properly, saving me all this hassle.

Re: [ANNOUNCE] ipset 6.11 released

[Jozsef Kadlecsik]

On Thu, 19 Jan 2012, Mr Dash Four wrote:

> > The userspace testing is passed to the kernel for execution. There's no
> > shadow or backup or whatever set in userspace. The sets exist in kernel
> > space and therefore all operations happen there.
> >   
> So, is the above doable in any shape or form or not?

Doable, but needs time and would involve adding the logic to auto-merge 
smaller subnets into larger ones and to make possible to delete subnets 
from larger networks.

I'm sure you are aware, this is a free time project.

> > Why do you need such tests at all?
> >   
> Various reasons. Two common uses (at least in my case) would be to test ip
> range against quite a large set of registered subnets (taken from the geoip
> database and sorted using my own criteria). The tested ip range is either
> candidate to 1). ban that network, in which case I do not want duplicates if
> the existing range is already there; or 2) include the ip range in a separate
> set, making sure that it is not already in the 'banned' set and also that it
> is not already included in the 'customer' set (which although limited, has
> quite significant number of set members - usually small subnets).
> 
> Because the ip range in ipset is not working correctly, I had to first
> manually go over the testing ranges. Later on I devised a small shell script
> doing what I just described in my previous post (quoted above), but it is
> quite ugly and inefficient - it would be much better if ip range testing in
> ipset was functioning properly, saving me all this hassle.

Sorry, for a while you have to use workarounds like you described.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.11 released

[Mr Dash Four]

> Doable, but needs time and would involve adding the logic to auto-merge 
> smaller subnets into larger ones and to make possible to delete subnets 
> from larger networks.
>   
Merging of ip ranges would be very difficult - I am having the same 
problem here as well. I use PostgreSQL as part of my setup and even 
though it has some nice capabilities to deal with "common" ip/range/net 
address operations, it is still very difficult, nigh-impossible to deal 
with ip range merges, simply because there are so many possible 
scenarios: any two ranges could be partially overlapping - like [ (] ), 
one range completely overlapping another - like [ () ], or two ranges 
"touching" one another - like [ ]( ) - that last one is very difficult 
to spot for merging and can't see it done very easily.

Just for testing ip range matches though, things could be a bit easier 
(pun intended).

If we assume that the whole ip address space (0.0.0.0-255.255.255.255, 
similar for ipv6 as well) occupies X number of bits (0...X-1), then each 
ip range (including a single ip address) could be represented by a 
starting bit number, length (in bits) and an end bit number (the latter 
is optional, but would ease ip range matching, if included). These 3 
components would have roughly the same memory footprint no matter what 
is the size of the actual ip range.

When adding such ip range in a set, I assume you just store their hash 
(to find them quickly when searching for exact set members). If you 
could add the above 3 elements, then matching ip ranges should be fairly 
easy: you take the starting bit from the ip range which needs to be 
tested and then isolate all those elements in the set which have their 
starting bit and ending bit in-between the starting bit of the testing 
range, i.e. test_start_bit between set_member_start_bit and 
set_member_end_bit, *or*, having their ending bit also between those 
ranges (i.e. test_end_bit between set_member_start_bit and 
srt_member_end_bit). You also need to discard duplicates, so that you 
don't test more than once.

Once you've isolated those potential matches, you can then construct a 
"partial" bitmap of these potential matches by using the 3 stored set 
member attributes - starting bit, length and end bit.

Then a simple bitwise "or" ("|") operation against each "partial" set 
member bitmap matched would suffice. By "partial" I mean applying the 
"or" only on the bit-range value we are interested in. This needs to be 
done recursively, whereby the result of the previous bitwise "or" 
operation should be applied to the partial bitmap of the next set 
member, so that you get partial matches accounted for.

If the end result after this leaves you with the exact bitmap of the ip 
range being tested, then there is a complete match. If you end up with 
something different, then there is either no or incomplete match. I hope 
I am making sense.

One very simple (and very crude) example to illustrate this: suppose the 
ip range we are interested in testing has a start bit value of 10 and 
its length is 16 (therefore the end bit = 25). Now, suppose we have 4 
set members with the following values stored in the set:

1. start-> 2, length-> 6
2. start-> 13, length-> 7
3. start-> 8, length-> 8
4. start-> 20, length-> 6

The "partial" bitmap of the ip range to be tested (stat bit = 10, end 
bit = 25) could be presented like this, after being "normalised" 
(position indicates bit number, starting from 0):

.........1111111111111111

Therefore, after initial scanning, we discard element 1 as it doesn't 
have its either start or end bit in the range we are interested in (10 - 
25) and that leaves us with the following set members:

.........0001111111000000
.......111111110000000000
.........0000000000111111

When we apply bitwise or ("|") on the range we are interested in (bits 
10-25) - recursively for each member left in the pot - we are left with 
a value of all 1's for the range of bits we are interested in (bits 
10-25), which is exactly the same as the ip range we are testing, so 
there is a complete match, otherwise there would either be no or 
incomplete match.

> I'm sure you are aware, this is a free time project.
>   
I am! I am also trying to help you with this as I would benefit from 
this bug being resolved.

> Sorry, for a while you have to use workarounds like you described.
>   
Pity that!

Re: [ANNOUNCE] ipset 6.11 released

[Amos Jeffries]

On 21/01/2012 5:45 a.m., Mr Dash Four wrote:
>
>> Doable, but needs time and would involve adding the logic to 
>> auto-merge smaller subnets into larger ones and to make possible to 
>> delete subnets from larger networks.
> Merging of ip ranges would be very difficult - I am having the same 
> problem here as well. I use PostgreSQL as part of my setup and even 
> though it has some nice capabilities to deal with "common" 
> ip/range/net address operations, it is still very difficult, 
> nigh-impossible to deal with ip range merges, simply because there are 
> so many possible scenarios: any two ranges could be partially 
> overlapping - like [ (] ), one range completely overlapping another - 
> like [ () ], or two ranges "touching" one another - like [ ]( ) - that 
> last one is very difficult to spot for merging and can't see it done 
> very easily.

Jozsef, have you had a look at how rbldnsd handles ranges? It has to 
deal with the same issues as ipset seems to be facing now, also at some 
high speed requirements. Maybe you could pull some inspiration out of that.

AYJ

Re: [ANNOUNCE] ipset 6.11 released

[Jozsef Kadlecsik]

On Sat, 21 Jan 2012, Amos Jeffries wrote:

> On 21/01/2012 5:45 a.m., Mr Dash Four wrote:
> > 
> > > Doable, but needs time and would involve adding the logic to auto-merge
> > > smaller subnets into larger ones and to make possible to delete subnets
> > > from larger networks.
> > Merging of ip ranges would be very difficult - I am having the same problem
> > here as well. I use PostgreSQL as part of my setup and even though it has
[...] 
> Jozsef, have you had a look at how rbldnsd handles ranges? It has to 
> deal with the same issues as ipset seems to be facing now, also at some 
> high speed requirements. Maybe you could pull some inspiration out of 
> that.

Thanks for the tip, I'll look into it.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary