[ANNOUNCE] ipset 6.5 released

[ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

Hi,

I'm happy to announce ipset 6.5. The list of changes are:

Kernel part changes:
 - Support range for IPv4 at adding/deleting elements for hash:*net* types
 - Set type support with multiple revisions added
 - Fix adding ranges to hash types

Userspace changes:
 - Support range for IPv4 at adding/deleting elements for hash:*net* types
 - Disable type revisions which are not supported both by the kernel and
   ipset
 - Update ipset help text to reflect SCTP and UDPLITE support
 - Ignore -n flag (list just setnames) when sets are to be saved

You can download the source code of ipset from:
        http://ipset.netfilter.org
        ftp://ftp.netfilter.org/pub/ipset/
        git://git.netfilter.org/ipset.git

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

>  - Support range for IPv4 at adding/deleting elements for hash:*net* types
>  - Set type support with multiple revisions added
>  - Fix adding ranges to hash types
>   
That's great news! Should I assume that the (more-than-double) hash bug 
I reported a while ago has also been fixed?

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Mon, 16 May 2011, Mr Dash Four wrote:

> >  - Support range for IPv4 at adding/deleting elements for hash:*net* types
> >  - Set type support with multiple revisions added
> >  - Fix adding ranges to hash types
> >   
> That's great news! Should I assume that the (more-than-double) hash bug I
> reported a while ago has also been fixed?

Yes, it was fixed, the last one ("Fix adding ranges to hash types") covers 
that.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

>> That's great news! Should I assume that the (more-than-double) hash bug I
>> reported a while ago has also been fixed?
>>     
>
> Yes, it was fixed, the last one ("Fix adding ranges to hash types") covers 
> that.
>   
Thank you Jozsef.

I have quickly tried the new version last night on a freshly compiled 
kernel (with ipset kernel modules included into the kernel - not as 
modules though) - the loading performance of "bulk" ranges is much, much 
better now, though I've got an error when trying to load a set which 
consisted of just 3 different subnet ranges (that set was using the 
"old" treemap syntax). When I have some more time - probably later this 
evening - I will post details of that error (I think it was to do with 
hashes again).

Also, I don't know whether you will be interested in this, but when I 
modified ipset 4.5 in order to split the kernel modules from the 
userspace tool and include these in two separate building streams for my 
systems, I managed to change ipset 4.5 to enable the use of 
autogen/automake tools and then the execution of "./configure" as part 
of building that package. As part of that process I introduced 2 extra 
"parameters" to it - "--hash-size" and "--max-elements". I could send 
you this patch to "refresh" the ipset 4.5 version or just have a look at 
it, if you so desire - just let me know. ipset 4.5 has served me very 
well since I started using it and was a vital contributor to the 
improved stability and performance on all my systems.

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Tue, 17 May 2011, Mr Dash Four wrote:

> I have quickly tried the new version last night on a freshly compiled kernel
> (with ipset kernel modules included into the kernel - not as modules though) -
> the loading performance of "bulk" ranges is much, much better now, though I've
> got an error when trying to load a set which consisted of just 3 different
> subnet ranges (that set was using the "old" treemap syntax). When I have some
> more time - probably later this evening - I will post details of that error (I
> think it was to do with hashes again).

Just send me the errors. The "treemap" backward compatibility support did 
not change between 6.4 and 6.5.
 
> Also, I don't know whether you will be interested in this, but when I modified
> ipset 4.5 in order to split the kernel modules from the userspace tool and
> include these in two separate building streams for my systems, I managed to
> change ipset 4.5 to enable the use of autogen/automake tools and then the
> execution of "./configure" as part of building that package. As part of that
> process I introduced 2 extra "parameters" to it - "--hash-size" and
> "--max-elements". I could send you this patch to "refresh" the ipset 4.5
> version or just have a look at it, if you so desire - just let me know. ipset
> 4.5 has served me very well since I started using it and was a vital
> contributor to the improved stability and performance on all my systems.

More control over the building process is good, so I'm interested in your 
patches.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Just send me the errors. The "treemap" backward compatibility support did 
> not change between 6.4 and 6.5.
>   
I'll prepare a test case and include the errors I get when I am back 
home later tonight. I'll do that then.

> More control over the building process is good, so I'm interested in your 
> patches.
>   
Will attach these together with the above. I was quite pleased with it 
as it enabled me to streamline the whole process (ipset 4.5 wasn't the 
only thing I was building), so you may find this quite useful if you 
decide to update v4.5.

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Will attach these together with the above. I was quite pleased with it 
> as it enabled me to streamline the whole process (ipset 4.5 wasn't the 
> only thing I was building), so you may find this quite useful if you 
> decide to update v4.5.
Here is the test case, I'll attach the rest privately to avoid the wrath 
of the mail list daemon.

[root@test1 ~]# ipset -N slow_mirrors iptreemap --gc 300
[root@test1 ~]# ipset -L slow_mirrors
Name: slow_mirrors
Type: hash:ip
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16512
References: 0
Members:
[root@test1 ~]# ipset -A slow_mirrors 212.219.56.128/26
[root@test1 ~]# ipset -A slow_mirrors 130.59.0.0/16
ipset v6.5: Hash is full, cannot add more elements
[root@xp1 ~]# ipset -L -terse slow_mirrors
Name: slow_mirrors
Type: hash:ip
Header: family inet hashsize 32768 maxelem 65536
Size in memory: 1006032
References: 0

The way I look at it, the "old" iptree(map) type sets should be 
converted to hash:net, not hash:ip to avoid this error.

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Tue, 17 May 2011, Mr Dash Four wrote:

> > Will attach these together with the above. I was quite pleased with it as it
> > enabled me to streamline the whole process (ipset 4.5 wasn't the only thing
> > I was building), so you may find this quite useful if you decide to update
> > v4.5.
> Here is the test case, I'll attach the rest privately to avoid the wrath of
> the mail list daemon.
> 
> [root@test1 ~]# ipset -N slow_mirrors iptreemap --gc 300
> [root@test1 ~]# ipset -L slow_mirrors
> Name: slow_mirrors
> Type: hash:ip
> Header: family inet hashsize 1024 maxelem 65536
> Size in memory: 16512
> References: 0
> Members:
> [root@test1 ~]# ipset -A slow_mirrors 212.219.56.128/26
> [root@test1 ~]# ipset -A slow_mirrors 130.59.0.0/16

The latter itself exhausts the maximal number of elements in the set, so 
the error message is normal.

> ipset v6.5: Hash is full, cannot add more elements
> [root@xp1 ~]# ipset -L -terse slow_mirrors
> Name: slow_mirrors
> Type: hash:ip
> Header: family inet hashsize 32768 maxelem 65536
> Size in memory: 1006032
> References: 0
> 
> The way I look at it, the "old" iptree(map) type sets should be converted to
> hash:net, not hash:ip to avoid this error.

That's too late, I can't change the mapping from hash:ip to hash:net.
(And if the mapping pointed to hash:net, you were surprised then that 
after adding say 192.68.0.0/24 to the set, you couldn't delete 192.68.0.1 
from it. :-)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> The latter itself exhausts the maximal number of elements in the set, so 
> the error message is normal.
>   
Is this a constraint coming from the "maxelem" value or the hash size? 
Could this be corrected by increasing either of these?

>> The way I look at it, the "old" iptree(map) type sets should be converted to
>> hash:net, not hash:ip to avoid this error.
>>     
>
> That's too late, I can't change the mapping from hash:ip to hash:net.
> (And if the mapping pointed to hash:net, you were surprised then that 
> after adding say 192.68.0.0/24 to the set, you couldn't delete 192.68.0.1 
> from it. :-)
>   
Nope. I am aware that deleting elements in 6.x should be done on the 
whole range when added, not just a single ip address (I am now clear as 
to why that happens and I think it makes sense in the way it is designed).

If you are to provide "backward" compatibility (or compatibility of any 
sort) with iptree(map) sets then you should make sure that at least 
members of that type of set could, at least, register without problem 
within the new set to which iptree(map) is converted. It seems to me 
that is not the case - with 6.5 at least. If it is difficult to maintain 
such compatibility you could do what you have already done with the 
"bindings" in previous versions of ipset - just drop it and ask users to 
switch to the new type. Frankly, it is not that difficult to switch over 
to the new type if one is aware of the implications, so it wouldn't be a 
big shock.

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Wed, 18 May 2011, Mr Dash Four wrote:

> > The latter itself exhausts the maximal number of elements in the set, so the
> > error message is normal.
> >   
> Is this a constraint coming from the "maxelem" value or the hash size? Could
> this be corrected by increasing either of these?

Yes, of course: use larger maxelem. The initial hash size value can also 
be increased, that way you can spare some internal resizings.
 
> > > The way I look at it, the "old" iptree(map) type sets should be converted
> > > to
> > > hash:net, not hash:ip to avoid this error.
> > >     
> > 
> > That's too late, I can't change the mapping from hash:ip to hash:net.
> > (And if the mapping pointed to hash:net, you were surprised then that after
> > adding say 192.68.0.0/24 to the set, you couldn't delete 192.68.0.1 from it.
> > :-)
> >   
> Nope. I am aware that deleting elements in 6.x should be done on the whole
> range when added, not just a single ip address (I am now clear as to why that
> happens and I think it makes sense in the way it is designed).
> 
> If you are to provide "backward" compatibility (or compatibility of any sort)
> with iptree(map) sets then you should make sure that at least members of that
> type of set could, at least, register without problem within the new set to
> which iptree(map) is converted. It seems to me that is not the case - with 6.5
> at least. If it is difficult to maintain such compatibility you could do what
> you have already done with the "bindings" in previous versions of ipset - just
> drop it and ask users to switch to the new type. Frankly, it is not that
> difficult to switch over to the new type if one is aware of the implications,
> so it wouldn't be a big shock.

I'll drop iptree(map) support in the next release ;-).

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Yes, it was fixed, the last one ("Fix adding ranges to hash types") covers 
> that.
>   
I disagree! I am still experiencing the same problems I've had with 
ipset 6.4, despite those being "fixed". When I try to load a set 
containing about 15k elements I get variations on the same type of error 
below:

ipset v6.5: Error in line 3556: Element cannot be added to the set: it's 
already added
ipset v6.5: Error in line 3653: Element cannot be added to the set: it's 
already added
ipset v6.5: Error in line 3340: Element cannot be added to the set: it's 
already added
ipset v6.5: Error in line 3266: Element cannot be added to the set: it's 
already added
ipset v6.5: Error in line 3653: Element cannot be added to the set: it's 
already added
ipset v6.5: Error in line 11187: Hash is full, cannot add more elements
ipset v6.5: Error in line 11182: Hash is full, cannot add more elements
ipset v6.5: Error in line 11185: Hash is full, cannot add more elements
ipset v6.5: Error in line 11186: Hash is full, cannot add more elements

The above happens when I try to use restore with the same file (i.e. the 
file contents does *not* change). On top of it all I also started 
getting this:

May 21 19:47:49 dmz1 kernel: [ 2103.322341] ipset[2827]: segfault at 0 
ip 004f6655 sp bf9125f0 error 4 in libmnl.so.0.0.1[4f4000+4000]
May 21 19:39:03 dmz1 kernel: [ 1576.402070] ipset[2777]: segfault at 0 
ip 006ea655 sp bfcfd900 error 4 in libmnl.so.0.0.1[6e8000+4000]
May 21 19:47:49 dmz1 kernel: [ 2103.322341] ipset[2827]: segfault at 0 
ip 004f6655 sp bf9125f0 error 4 in libmnl.so.0.0.1[4f4000+4000]
May 21 19:57:32 dmz1 kernel: [ 2685.725299] ipset[2870]: segfault at 0 
ip 00339655 sp bf8bac90 error 4 in libmnl.so.0.0.1[337000+4000]
May 21 20:01:48 dmz1 kernel: [ 2942.232408] ipset[2913]: segfault at 0 
ip 00baa655 sp bfa25d80 error 4 in libmnl.so.0.0.1[ba8000+4000]
May 21 20:09:18 dmz1 kernel: [ 3392.073377] ipset[2928]: segfault at 0 
ip 00470655 sp bfcb43c0 error 4 in libmnl.so.0.0.1[46e000+4000]
May 21 20:10:57 dmz1 kernel: [ 3490.388755] ipset[2941]: segfault at 0 
ip 00b8a655 sp bffb92c0 error 4 in libmnl.so.0.0.1[b88000+4000]
May 21 20:12:36 dmz1 kernel: [ 3589.622479] ipset[2954]: segfault at 0 
ip 009bf655 sp bf8fef80 error 4 in libmnl.so.0.0.1[9bd000+4000]
May 21 20:13:21 dmz1 kernel: [ 3634.433419] ipset[2966]: segfault at 0 
ip 00d11655 sp bf9ae7a0 error 4 in libmnl.so.0.0.1[d0f000+4000]
May 21 20:13:28 dmz1 kernel: [ 3641.484628] ipset[2977]: segfault at 0 
ip 005e7655 sp bfedb090 error 4 in libmnl.so.0.0.1[5e5000+4000]
May 21 20:13:42 dmz1 kernel: [ 3656.205294] ipset[2988]: segfault at 0 
ip 009cb655 sp bfae5b80 error 4 in libmnl.so.0.0.1[9c9000+4000]
May 21 20:15:31 dmz1 kernel: [ 3765.137306] ipset[3001]: segfault at 0 
ip 0019e655 sp bfe3fff0 error 4 in libmnl.so.0.0.1[19c000+4000]

libmnl is 1.0.1. I have then tried to play a bit with the hashsize 
value, but gave up when I reached 16777216! ipset though was still 
insisting that the hash size is "full":

maxelem 15000 hashsize 4194304
ipset v6.5: Error in line 11191: Hash is full, cannot add more elements
maxelem 15000 hashsize 8388608
ipset v6.5: Error in line 11191: Hash is full, cannot add more elements
maxelem 15000 hashsize 16777216
ipset v6.5: Error in line 11191: Hash is full, cannot add more elements

-bash-4.1# ipset -L -terse test
Name: test
Type: hash:net
Header: family inet hashsize 16777216 maxelem 15000 timeout 0
Size in memory: 134937948
References: 0

So, 134MB of RAM is not enough for a set consisting of 15k members? 
Right! I am reverting to 4.5 again as this is crazy!

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Mr Dash Four wrote:

> > Yes, it was fixed, the last one ("Fix adding ranges to hash types") covers
> > that.
> >   
> I disagree! I am still experiencing the same problems I've had with ipset 6.4,
> despite those being "fixed". When I try to load a set containing about 15k
> elements I get variations on the same type of error below:
> 
> ipset v6.5: Error in line 3556: Element cannot be added to the set: it's
> already added

Send me the input you used so that I could reproduce it.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Mr Dash Four wrote:

> gave up when I reached 16777216! ipset though was still insisting that the
> hash size is "full":
> 
> maxelem 15000 hashsize 4194304

When you asked about maxelem and hashsize, I wrote:

> > Is this a constraint coming from the "maxelem" value or the hash size? 
> > Could this be corrected by increasing either of these?

> Yes, of course: use larger maxelem. The initial hash size value can also 
> be increased, that way you can spare some internal resizings.

You still have "maxelem 15000". That means there can be at the maximum 
15000 element in the hash, and in another words, ipset regards the set 
full when its elem count reaches 15000.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> You still have "maxelem 15000". That means there can be at the maximum 
> 15000 element in the hash, and in another words, ipset regards the set 
> full when its elem count reaches 15000.
>   
I have no disagreements over that at all. The set in question contains 
just over 11k members, the maxelem value is 15k, so plenty to spare. My 
grief is with the (seemingly) random error messages and failures I am 
getting - just like in 6.4, plus the new added segfault error with 
libmnl as of ipset 6.5. The memory isn't an issue and neither is the 
hashsize or the maxelem values.

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Mr Dash Four wrote:

> > Yes, it was fixed, the last one ("Fix adding ranges to hash types") covers
> > that.
> >   
> I disagree! I am still experiencing the same problems I've had with ipset 6.4,
> despite those being "fixed". When I try to load a set containing about 15k
> elements I get variations on the same type of error below:
> 
> ipset v6.5: Error in line 3556: Element cannot be added to the set: it's
> already added

You used the timeout variant of the hash type, and a mistypeing slipped 
into the patch titled "Fix adding ranges to hash types", which made the 
patch ineffective for the timeout variants :-(. Please try the patch 
below: apply to the source tree of the ipset package, recompile the kernel 
modules, install them, remove from the running kernel and try again.

> The above happens when I try to use restore with the same file (i.e. the file
> contents does *not* change). On top of it all I also started getting this:
> 
> May 21 19:47:49 dmz1 kernel: [ 2103.322341] ipset[2827]: segfault at 0 ip
> 004f6655 sp bf9125f0 error 4 in libmnl.so.0.0.1[4f4000+4000]

I could not get such segfaults at all.

> libmnl is 1.0.1. I have then tried to play a bit with the hashsize value, but
> gave up when I reached 16777216! ipset though was still insisting that the
> hash size is "full":
> 
> maxelem 15000 hashsize 4194304

As I wrote previously, that is because you reached the limit of the 
maximal elements in the hash. Increase maxelem, it's a fixed parameter of 
the hash. The hashsize parameter is an initial value and auto-tuned 
(incremented) as required.

Best regards,
Jozsef

diff --git a/kernel/include/linux/netfilter/ipset/ip_set_ahash.h b/kernel/include/linux/netfilter/ipset/ip_set_ahash.h
index 905e2ac..a099d40 100644
--- a/kernel/include/linux/netfilter/ipset/ip_set_ahash.h
+++ b/kernel/include/linux/netfilter/ipset/ip_set_ahash.h
@@ -835,7 +835,7 @@ type_pf_tadd(struct ip_set *set, void *value, u32 timeout, u32 flags)
 	}
 	ret = type_pf_elem_tadd(n, d, timeout);
 	if (ret != 0) {
-		if (ret == -EEXIST)
+		if (ret == -EAGAIN)
 			type_pf_data_next(h, d);
 		goto out;
 	}

-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Jan Engelhardt]

On Saturday 2011-05-21 23:31, Jozsef Kadlecsik wrote:

>On Sat, 21 May 2011, Mr Dash Four wrote:
>
>> The above happens when I try to use restore with the same file (i.e. the file
>> contents does *not* change). On top of it all I also started getting this:
>> 
>> May 21 19:47:49 dmz1 kernel: [ 2103.322341] ipset[2827]: segfault at 0 ip
>> 004f6655 sp bf9125f0 error 4 in libmnl.so.0.0.1[4f4000+4000]
>
>I could not get such segfaults at all.

Nevertheless, I would recommend to mr-4 to have a ipset and libmnl 
compiled with debug info, point to it via LD_LIBRARY_PATH, and then feed 
it through valgrind with the same input. That may give some more 
insight.

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> You used the timeout variant of the hash type, and a mistypeing slipped 
> into the patch titled "Fix adding ranges to hash types", which made the 
> patch ineffective for the timeout variants :-(.
I am not sure what you mean by the above!

>  Please try the patch 
> below: apply to the source tree of the ipset package, recompile the kernel 
> modules, install them, remove from the running kernel and try again.
>   
I'll do it in the coming days as I ran out of time and already reverted 
to ipset 4.5 as I don't have any more time to play with this.

>> libmnl is 1.0.1. I have then tried to play a bit with the hashsize value, but
>> gave up when I reached 16777216! ipset though was still insisting that the
>> hash size is "full":
>>
>> maxelem 15000 hashsize 4194304
>>     
>
> As I wrote previously, that is because you reached the limit of the 
> maximal elements in the hash. Increase maxelem, it's a fixed parameter of 
> the hash. The hashsize parameter is an initial value and auto-tuned 
> (incremented) as required.
>   
Jozsef, the set I was referring to has just over 11 thousand members, 
where the maxelem value for that set was set at 15 thousand - I thought 
I have made that pretty clear, but I obviously haven't!

Look at the files I sent you and let me know in which set I have 
specified maxelem value which is less than the number of members I am 
trying to add? My guess is, there are none!

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

>> I could not get such segfaults at all.
>>     
>
> Nevertheless, I would recommend to mr-4 to have a ipset and libmnl 
> compiled with debug info, point to it via LD_LIBRARY_PATH, and then feed 
> it through valgrind with the same input. That may give some more 
> insight.
>   
I am willing to help if you let me know what I need to do, though this 
will be done in the coming days as I have other things I need to take 
care of and ran out of time, unfortunately.

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Nevertheless, I would recommend to mr-4 to have a ipset and libmnl 
> compiled with debug info, point to it via LD_LIBRARY_PATH, and then feed 
> it through valgrind with the same input. That may give some more 
> insight.
>   
Oh, by the way, forgot to add this - there is a bug in xtables-addon I 
ran into while I was dealing with all this - if I have ipv6 compiled as 
a *module* when I try to install xtables-addons (via kickstart) I get 
these errors:

WARNING: 
/lib/modules/2.6.35.13-91.fc13.i686/extra/xtables-addons/xt_RAWNAT.ko 
needs unknown symbol ipv6_find_hdr
WARNING: 
/lib/modules/2.6.35.13-91.fc13.i686/extra/xtables-addons/xt_SYSRQ.ko 
needs unknown symbol ipv6_find_hdr
WARNING: 
/lib/modules/2.6.35.13-91.fc13.i686/extra/xtables-addons/ip6table_rawpost.ko 
needs unknown symbol ip6t_unregister_table

I managed to fix the "ip6table_rawpost.ko" error - the problem was with 
the extensions/Kbuild file - it needed another "ifneq 
(${CONFIG_IPV6_MODULE},)" "endif" block after "ifneq (${CONFIG_IPV6},)".

I saw in xt_RAWNAT.c that there is preprocessor directive which was like 
"#if defined(CONFIG_IPV6) || defined(CONFIG_IPV6_MODULE)" to separate 
the ipv6 code from the "normal" one, but it didn't work properly for 
some reason. When I forcefully removed the ipv6 code blocks from both 
xt_RAWNAT.c and xt_SYSRQ.c everything was fine, so I think you need to 
look at this!

Re: [ANNOUNCE] ipset 6.5 released

[Jan Engelhardt]

On Saturday 2011-05-21 23:41, Mr Dash Four wrote:

>
>>> I could not get such segfaults at all.
>>>    
>>
>> Nevertheless, I would recommend to mr-4 to have a ipset and libmnl compiled
>> with debug info, point to it via LD_LIBRARY_PATH, and then feed it through
>> valgrind with the same input. That may give some more insight.
>>  
> I am willing to help if you let me know what I need to do, though this will be
> done in the coming days as I have other things I need to take care of and ran
> out of time, unfortunately.

Get libmnl source (either git or tarball), and get ipset source 
likewise, and for each, ./configure CFLAGS="-O0 -ggdb3" and make - no 
install.

Subsequently (substitute by appropriate paths/actions):

LD_LIBRARY_PATH=$HOME/src/libmnl/src/.libs \
libtool --mode=execute valgrind $HOME/src/ipset/src/ipset $action

Reasonable recipe for any program :)

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Kernel part changes:
>  - Support range for IPv4 at adding/deleting elements for hash:*net* types
>  - Set type support with multiple revisions added
>  - Fix adding ranges to hash types
>   
Forgot one more thing: I was able to compile all ipset 4.5 kernel 
modules into the kernel itself (i.e. all ipset kernel module options set 
to "y" as oppose to "m"). I am, however, not able to do that in ipset 
6.5 - I can only compile them as modules (i.e. I only have two choices 
and that is either "m" to include them as modules, or not compile them 
at all). Is there a reason for this or is this another bug?

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Mr Dash Four wrote:

> > > gave up when I reached 16777216! ipset though was still insisting that the
> > > hash size is "full":
> > > 
> > > maxelem 15000 hashsize 4194304
> > >     
> > 
> > As I wrote previously, that is because you reached the limit of the maximal
> > elements in the hash. Increase maxelem, it's a fixed parameter of the hash.
> > The hashsize parameter is an initial value and auto-tuned (incremented) as
> > required.
> >   
> Jozsef, the set I was referring to has just over 11 thousand members, where
> the maxelem value for that set was set at 15 thousand - I thought I have made
> that pretty clear, but I obviously haven't!
> 
> Look at the files I sent you and let me know in which set I have specified
> maxelem value which is less than the number of members I am trying to add? My
> guess is, there are none!

Your first set named 'test_ips' consisting of 11670 ranges can only be 
converted to 15613 networks, i.e. members. Not all ranges can be expressed 
with a single network.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Mr Dash Four wrote:

> > Kernel part changes:
> >  - Support range for IPv4 at adding/deleting elements for hash:*net* types
> >  - Set type support with multiple revisions added
> >  - Fix adding ranges to hash types
> >   
> Forgot one more thing: I was able to compile all ipset 4.5 kernel modules into
> the kernel itself (i.e. all ipset kernel module options set to "y" as oppose
> to "m"). I am, however, not able to do that in ipset 6.5 - I can only compile
> them as modules (i.e. I only have two choices and that is either "m" to
> include them as modules, or not compile them at all). Is there a reason for
> this or is this another bug?

The ipset-4.x packages support patching the kernel source tree, and in the 
kernel tree you can select module or static compiling. The ipset-6.x 
packages support only compiling as external modules.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sun, 22 May 2011, Jozsef Kadlecsik wrote:

> On Sat, 21 May 2011, Mr Dash Four wrote:
> 
> > > > gave up when I reached 16777216! ipset though was still insisting that the
> > > > hash size is "full":
> > > > 
> > > > maxelem 15000 hashsize 4194304
> > > >     
> > > 
> > > As I wrote previously, that is because you reached the limit of the maximal
> > > elements in the hash. Increase maxelem, it's a fixed parameter of the hash.
> > > The hashsize parameter is an initial value and auto-tuned (incremented) as
> > > required.
> > >   
> > Jozsef, the set I was referring to has just over 11 thousand members, where
> > the maxelem value for that set was set at 15 thousand - I thought I have made
> > that pretty clear, but I obviously haven't!
> > 
> > Look at the files I sent you and let me know in which set I have specified
> > maxelem value which is less than the number of members I am trying to add? My
> > guess is, there are none!
> 
> Your first set named 'test_ips' consisting of 11670 ranges can only be 
> converted to 15613 networks, i.e. members. Not all ranges can be expressed 
> with a single network.

Just look at the firs range in that file: with the first octet anonymized, 
it's

a test 10.0.1.0-10.0.3.255

However that's identical with two networks, 10.0.1.0/24 and 10.0.2.0/23, 
thus two set members. So the command above is equivalent with

a test 10.0.1.0/24
a test 10.0.2.0/23

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> The ipset-4.x packages support patching the kernel source tree, and in the 
> kernel tree you can select module or static compiling. The ipset-6.x 
> packages support only compiling as external modules.
Fair enough, is there any reason why I can't include them as part of the kernel (i.e. static) as I used to with 4.5?

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Your first set named 'test_ips' consisting of 11670 ranges can only be 
> converted to 15613 networks, i.e. members. Not all ranges can be expressed 
> with a single network.
Oh, sorry, my bad! I forgot about this (new) aspect of it all.

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Just look at the firs range in that file: with the first octet anonymized, 
> it's
> 
> a test 10.0.1.0-10.0.3.255
> 
> However that's identical with two networks, 10.0.1.0/24 and 10.0.2.0/23, 
> thus two set members. So the command above is equivalent with
> 
> a test 10.0.1.0/24
> a test 10.0.2.0/23
Yeah, I understand it now - I think I need to allow for some more leverage when defining maxelem. Thanks.

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Mr Dash Four wrote:

> > The ipset-4.x packages support patching the kernel source tree, and in the 
> > kernel tree you can select module or static compiling. The ipset-6.x 
> > packages support only compiling as external modules.
> Fair enough, is there any reason why I can't include them as part of the 
> kernel (i.e. static) as I used to with 4.5?

I don't really understand what you mean here, sorry.

If you want the kernel part of the ipset package compiled statically, then 
that's not supported out of the box. Manually doable: get a kernel tree 
where ipset already included, copy over all *.[ch] files from 
ipset-6.x/kernel/ to the corresponding kernel subtrees and then continue 
configuring/compiling in the kernel tree.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

>> Fair enough, is there any reason why I can't include them as part of the 
>> kernel (i.e. static) as I used to with 4.5?
> 
> I don't really understand what you mean here, sorry.
> 
> If you want the kernel part of the ipset package compiled statically, then 
> that's not supported out of the box. Manually doable: get a kernel tree 
> where ipset already included, copy over all *.[ch] files from 
> ipset-6.x/kernel/ to the corresponding kernel subtrees and then continue 
> configuring/compiling in the kernel tree.
OK, as you know I compile ipset - userspace and kernel modules - in two separate building streams. The kernel part of ipset I compile as part of the kernel itself - via the make oldconfig, make menuconfig or by simply copying already pre-defined .config file (I obviously apply the ipset 6.5 patches as well - as you know).

With ipset 4.5 I was able to include all ipset "modules" as part of the kernel (i.e. all ipset-related options in my .config were "y" as oppose to "m").

In ipset 6.5 I cannot do that - the only choice I seem to have is "m" or not include them at all. So, I was wondering why that is, that's all?

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Jan Engelhardt wrote:

> On Saturday 2011-05-21 23:41, Mr Dash Four wrote:
> 
> >>> I could not get such segfaults at all.
> >>>    
> >>
> >> Nevertheless, I would recommend to mr-4 to have a ipset and libmnl compiled
> >> with debug info, point to it via LD_LIBRARY_PATH, and then feed it through
> >> valgrind with the same input. That may give some more insight.
> >>  
> > I am willing to help if you let me know what I need to do, though this will be
> > done in the coming days as I have other things I need to take care of and ran
> > out of time, unfortunately.
> 
> Get libmnl source (either git or tarball), and get ipset source 
> likewise, and for each, ./configure CFLAGS="-O0 -ggdb3" and make - no 
> install.
> 
> Subsequently (substitute by appropriate paths/actions):
> 
> LD_LIBRARY_PATH=$HOME/src/libmnl/src/.libs \
> libtool --mode=execute valgrind $HOME/src/ipset/src/ipset $action
> 
> Reasonable recipe for any program :)

With the original ipset-6.5 and restoring the privately sent test_ips it 
produces:

==2423== Memcheck, a memory error detector.
==2423== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al.
==2423== Using LibVEX rev 1884, a library for dynamic binary translation.
==2423== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP.
==2423== Using valgrind-3.4.1-Debian, a dynamic binary instrumentation framework.
==2423== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al.
==2423== For more details, rerun with: -v
==2423== 
==2423== Syscall param socketcall.sendto(msg) points to uninitialised byte(s)
==2423==    at 0x5118183: sendto (in /lib/libc-2.9.so)
==2423==    by 0x4E2DBDB: mnl_socket_sendto (socket.c:183)
==2423==    by 0x40CD06: ipset_mnl_query (mnl.c:95)
==2423==    by 0x40B032: build_send_private_msg (session.c:1567)
==2423==    by 0x40BAA1: ipset_cmd (session.c:1843)
==2423==    by 0x40C307: create_type_get (types.c:239)
==2423==    by 0x40C7DD: ipset_type_get (types.c:382)
==2423==    by 0x4067CA: ipset_parse_typename (parse.c:1389)
==2423==    by 0x40327E: parse_commandline (ipset.c:618)
==2423==    by 0x402414: restore (ipset.c:197)
==2423==    by 0x4033EF: parse_commandline (ipset.c:667)
==2423==    by 0x403695: main (ipset.c:742)
==2423==  Address 0x7ff000269 is on thread 1's stack
ipset v6.5: Error in line 2807: Element cannot be added to the set: it's already added
==2423== 
==2423== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 8 from 1)
==2423== malloc/free: in use at exit: 0 bytes in 0 blocks.
==2423== malloc/free: 17,327 allocs, 17,327 frees, 610,049 bytes allocated.
==2423== For counts of detected errors, rerun with: -v
==2423== Use --track-origins=yes to see where uninitialised values come from
==2423== All heap blocks were freed -- no leaks are possible.

The same one with the patched ipset-6.5:

==2229== Memcheck, a memory error detector.
==2229== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al.
==2229== Using LibVEX rev 1884, a library for dynamic binary translation.
==2229== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP.
==2229== Using valgrind-3.4.1-Debian, a dynamic binary instrumentation framework.
==2229== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al.
==2229== For more details, rerun with: -v
==2229== 
==2229== Syscall param socketcall.sendto(msg) points to uninitialised byte(s)
==2229==    at 0x5118183: sendto (in /lib/libc-2.9.so)
==2229==    by 0x4E2DBDB: mnl_socket_sendto (socket.c:183)
==2229==    by 0x40CD06: ipset_mnl_query (mnl.c:95)
==2229==    by 0x40B032: build_send_private_msg (session.c:1567)
==2229==    by 0x40BAA1: ipset_cmd (session.c:1843)
==2229==    by 0x40C307: create_type_get (types.c:239)
==2229==    by 0x40C7DD: ipset_type_get (types.c:382)
==2229==    by 0x4067CA: ipset_parse_typename (parse.c:1389)
==2229==    by 0x40327E: parse_commandline (ipset.c:618)
==2229==    by 0x402414: restore (ipset.c:197)
==2229==    by 0x4033EF: parse_commandline (ipset.c:667)
==2229==    by 0x403695: main (ipset.c:742)
==2229==  Address 0x7ff000269 is on thread 1's stack
ipset v6.5: Error in line 11191: Hash is full, cannot add more elements
==2229== 
==2229== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 8 from 1)
==2229== malloc/free: in use at exit: 0 bytes in 0 blocks.
==2229== malloc/free: 67,277 allocs, 67,277 frees, 2,385,155 bytes allocated.
==2229== For counts of detected errors, rerun with: -v
==2229== Use --track-origins=yes to see where uninitialised values come from
==2229== All heap blocks were freed -- no leaks are possible.

The uninitialized bytes are OK and there is no memory leak or corruption.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sat, 21 May 2011, Mr Dash Four wrote:

> >> Fair enough, is there any reason why I can't include them as part of the 
> >> kernel (i.e. static) as I used to with 4.5?
> > 
> > I don't really understand what you mean here, sorry.
> > 
> > If you want the kernel part of the ipset package compiled statically, then 
> > that's not supported out of the box. Manually doable: get a kernel tree 
> > where ipset already included, copy over all *.[ch] files from 
> > ipset-6.x/kernel/ to the corresponding kernel subtrees and then continue 
> > configuring/compiling in the kernel tree.
> OK, as you know I compile ipset - userspace and kernel modules - in two 
> separate building streams. The kernel part of ipset I compile as part of 
> the kernel itself - via the make oldconfig, make menuconfig or by simply 
> copying already pre-defined .config file (I obviously apply the ipset 
> 6.5 patches as well - as you know).

I haven't time yet to look at the packages you sent me, so I don't know 
the exact details of your build process.

> With ipset 4.5 I was able to include all ipset "modules" as part of the 
> kernel (i.e. all ipset-related options in my .config were "y" as oppose 
> to "m").
> 
> In ipset 6.5 I cannot do that - the only choice I seem to have is "m" or 
> not include them at all. So, I was wondering why that is, that's all?

I think you have got NETFILTER_NETLINK_QUEUE, NETFILTER_NETLINK_LOG or 
NF_CT_NETLINK selected as module, and in consequence NETFILTER_NETLINK as 
module too. And because the ipset kernel modules depend on 
NETFILTER_NETLINK, then those can be selected as modules only too.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> I think you have got NETFILTER_NETLINK_QUEUE, NETFILTER_NETLINK_LOG or 
> NF_CT_NETLINK selected as module, and in consequence NETFILTER_NETLINK as 
> module too. And because the ipset kernel modules depend on 
> NETFILTER_NETLINK, then those can be selected as modules only too.
I have them all as modules (i.e. "m"), *but* that is the case with 4.5 as well - I just checked my 4.5 .config, so there is not difference in that respect.

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> The uninitialized bytes are OK and there is no memory leak or corruption.
>   
I am able to now reproduce the error, having applied your patch Jozsef, 
rebuilt the kernel, updated the whole image on that machine and 
rebooted. This is what happens:

All sets are now loaded, *except* the last file where I get the 
segmentation fault error I described earlier, i.e, something like this:

May 22 01:25:20 dmz1 kernel: [ 1361.321376] ipset[2819]: segfault at 0 
ip 00e26655 sp bfdad710 error 4 in libmnl.so.0.0.1[e24000+4000]
May 22 01:26:39 dmz1 kernel: [ 1440.120529] ipset[2841]: segfault at 0 
ip 002d7655 sp bfbe7b10 error 4 in libmnl.so.0.0.1[2d5000+4000]

I started to narrow this down and the first culprit seems to be the 
following 2 statements:

n priviledged-ports bitmap:port range 1-1023 timeout 0
a priviledged-ports 1-1023

When I removed the above 2 statements I am then getting this:

ipset v6.5: Error in line 4: Internal protocol error

No segmentation fault! I narrowed the latest error down to the following 
2 statements:

n test-net bitmap:ip range 10.1.1.172/29 timeout 0
a test-net 10.1.1.172/29

I then removed those as well. Tried again - segmentation fault error. I 
then removed the following 3 statements, which were next in line:

n test-ports bitmap:port range 12770-19999 timeout 0
a test-ports 12770-12784
a test-ports 19999

Next, another segmentation fault error, it turns out the culprit this 
time was 2 statements which were further down the line from this file:

n test-port bitmap:port range 29950-29950 timeout 0
a test-port 29950

Removed these and guess what - segmentation fault error again, at which 
point I bloody gave up! ipset 4.5 it is!

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sun, 22 May 2011, Mr Dash Four wrote:

> > I think you have got NETFILTER_NETLINK_QUEUE, NETFILTER_NETLINK_LOG or 
> > NF_CT_NETLINK selected as module, and in consequence NETFILTER_NETLINK as 
> > module too. And because the ipset kernel modules depend on 
> > NETFILTER_NETLINK, then those can be selected as modules only too.
> I have them all as modules (i.e. "m"), *but* that is the case with 4.5 
> as well - I just checked my 4.5 .config, so there is not difference in 
> that respect.

ipset-4.x did not depend on NETFILTER_NETLINK, while ipset-6.x does.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sun, 22 May 2011, Mr Dash Four wrote:

> > The uninitialized bytes are OK and there is no memory leak or corruption.
> >   
> I am able to now reproduce the error, having applied your patch Jozsef,
> rebuilt the kernel, updated the whole image on that machine and rebooted. This
> is what happens:
> 
> All sets are now loaded, *except* the last file where I get the segmentation
> fault error I described earlier, i.e, something like this:
> 
> May 22 01:25:20 dmz1 kernel: [ 1361.321376] ipset[2819]: segfault at 0 ip
> 00e26655 sp bfdad710 error 4 in libmnl.so.0.0.1[e24000+4000]
> May 22 01:26:39 dmz1 kernel: [ 1440.120529] ipset[2841]: segfault at 0 ip
> 002d7655 sp bfbe7b10 error 4 in libmnl.so.0.0.1[2d5000+4000]
> 
> I started to narrow this down and the first culprit seems to be the following
> 2 statements:
> 
> n priviledged-ports bitmap:port range 1-1023 timeout 0
> a priviledged-ports 1-1023
> 
> When I removed the above 2 statements I am then getting this:
> 
> ipset v6.5: Error in line 4: Internal protocol error
> 
> No segmentation fault! I narrowed the latest error down to the following 2
> statements:
> 
> n test-net bitmap:ip range 10.1.1.172/29 timeout 0
> a test-net 10.1.1.172/29
> 
> I then removed those as well. Tried again - segmentation fault error. I then
> removed the following 3 statements, which were next in line:
> 
> n test-ports bitmap:port range 12770-19999 timeout 0
> a test-ports 12770-12784
> a test-ports 19999
> 
> Next, another segmentation fault error, it turns out the culprit this time was
> 2 statements which were further down the line from this file:
> 
> n test-port bitmap:port range 29950-29950 timeout 0
> a test-port 29950
> 
> Removed these and guess what - segmentation fault error again, at which point
> I bloody gave up! ipset 4.5 it is!

If you want me to be able to reproduce and fix the issue, please send me 
the full data you used. I assumed the restore files you sent me contained 
the case where the segfaults popped up. Apparently it's not, because 
there's no such commands in them.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sun, 22 May 2011, Jozsef Kadlecsik wrote:

> If you want me to be able to reproduce and fix the issue, please send me 
> the full data you used. I assumed the restore files you sent me contained 
> the case where the segfaults popped up. Apparently it's not, because 
> there's no such commands in them.

And send me the architecture you run, maybe it's an architecture-dependent 
issue. My test environment is x86_64.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> If you want me to be able to reproduce and fix the issue, please send me 
> the full data you used. I assumed the restore files you sent me contained 
> the case where the segfaults popped up. Apparently it's not, because 
> there's no such commands in them.
>   
Try this and let me know how far do you get:

n priviledged-ports bitmap:port range 1-1023 timeout 0
a priviledged-ports 1-1023
n test-net bitmap:ip range 10.1.1.172/29 timeout 0
a test-net 10.1.1.172/29
n test-ports bitmap:port range 12770-19999 timeout 0
a test-ports 12770-12784
a test-ports 19999
n test-port bitmap:port range 29950-29950 timeout 0
a test-port 29950
COMMIT

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> And send me the architecture you run, maybe it's an architecture-dependent 
> issue. My test environment is x86_64.
>   
No, it is not. I tried this on i686 (P2 as well as P4) and x86_64 
(Core2) systems with the same effect and errors, so it is not 
arch-dependent error.

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Get libmnl source (either git or tarball), and get ipset source 
> likewise, and for each, ./configure CFLAGS="-O0 -ggdb3" and make - no 
> install.
>
> Subsequently (substitute by appropriate paths/actions):
>
> LD_LIBRARY_PATH=$HOME/src/libmnl/src/.libs \
> libtool --mode=execute valgrind $HOME/src/ipset/src/ipset $action
>
> Reasonable recipe for any program :)
>   
[me@test1 src]$ cat ipset-6.5-segfault.ips
n priviledged-ports bitmap:port range 1-1023 timeout 0
a priviledged-ports 1-1023
n test-net bitmap:ip range 10.1.1.172/29 timeout 0
a test-net 10.1.1.172/29
n test-ports bitmap:port range 12770-19999 timeout 0
a test-ports 12770-12784
a test-ports 19999
n test-port bitmap:port range 29950-29950 timeout 0
a test-port 29950
COMMIT
[me@test1 src]$ LD_LIBRARY_PATH=/home/me/src/libmnl-1.0.1/src/.libs 
libtool --mode=execute valgrind --track-origins=yes -v 
/home/me/src/ipset-6.5/src/ipset restore < ipset-6.5-segfault.ips
==10502== Memcheck, a memory error detector
==10502== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==10502== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==10502== Command: /home/me/src/ipset-6.5/src/ipset restore
==10502==
--10502-- Valgrind options:
--10502--    --track-origins=yes
--10502--    -v
--10502-- Contents of /proc/version:
--10502--   Linux version 2.6.35.13-91.fc13.x86_64 (me@test1.my.net) 
(gcc version 4.4.5 20101112 (Red Hat 4.4.5-2) (GCC) ) #1 SMP Sat May 21 
02:33:34 BST 2011
--10502-- Arch and hwcaps: AMD64, amd64-sse3-cx16
--10502-- Page sizes: currently 4096, max supported 4096
--10502-- Valgrind library directory: /usr/lib64/valgrind
--10502-- Reading syms from /home/me/src/ipset-6.5/src/ipset (0x400000)
--10502-- Reading syms from /usr/lib64/valgrind/memcheck-amd64-linux 
(0x38000000)
--10502--    object doesn't have a dynamic symbol table
--10502-- Reading syms from /lib64/ld-2.12.2.so (0x3c86600000)
--10502-- Reading suppressions file: /usr/lib64/valgrind/default.supp
--10502-- REDIR: 0x3c86617350 (strlen) redirected to 0x3803f6f7 
(vgPlain_amd64_linux_REDIR_FOR_strlen)
--10502-- Reading syms from 
/usr/lib64/valgrind/vgpreload_core-amd64-linux.so (0x4801000)
--10502-- Reading syms from 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so (0x4a02000)
==10502== WARNING: new redirection conflicts with existing -- ignoring it
--10502--     new: 0x3c86617350 (strlen              ) R-> 0x04a06340 strlen
--10502-- REDIR: 0x3c866171c0 (index) redirected to 0x4a05f80 (index)
--10502-- REDIR: 0x3c86617240 (strcmp) redirected to 0x4a06900 (strcmp)
--10502-- Reading syms from 
/home/me/src/libmnl-1.0.1/src/.libs/libmnl.so.0.0.1 (0x4c0a000)
--10502-- Reading syms from /lib64/libc-2.12.2.so (0x3c86a00000)
--10502-- REDIR: 0x3c86a815a0 (__GI_strrchr) redirected to 0x4a05e00 
(__GI_strrchr)
--10502-- REDIR: 0x3c86a7fab0 (strlen) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
==10502== WARNING: new redirection conflicts with existing -- ignoring it
--10502--     new: 0x3c86a7fae0 (__GI_strlen         ) R-> 0x04a062e0 strlen
--10502-- REDIR: 0x3c86a7fae0 (__GI_strlen) redirected to 0x4a06300 
(__GI_strlen)
--10502-- REDIR: 0x3c86a7e070 (strcmp) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
--10502-- REDIR: 0x3c86b26cc0 (__strcmp_ssse3) redirected to 0x4a06860 
(strcmp)
--10502-- REDIR: 0x3c86a79180 (calloc) redirected to 0x4a043d3 (calloc)
--10502-- REDIR: 0x3c86a7fcb0 (strncmp) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
--10502-- REDIR: 0x3c86b27f20 (__strncmp_ssse3) redirected to 0x4a067a0 
(strncmp)
--10502-- REDIR: 0x3c86a81cb0 (memchr) redirected to 0x4a069a0 (memchr)
--10502-- REDIR: 0x3c86a83730 (memcpy) redirected to 0x4a069d0 (memcpy)
--10502-- REDIR: 0x3c86a81540 (strncpy) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
--10502-- REDIR: 0x3c86b24140 (???) redirected to 0x4a06500 (strncpy)
--10502-- REDIR: 0xffffffffff600400 (???) redirected to 0x3803f6ed 
(vgPlain_amd64_linux_REDIR_FOR_vtime)
--10502-- REDIR: 0x3c86a82320 (memset) redirected to 0x4a07540 (memset)
==10502== Syscall param socketcall.sendto(msg) points to uninitialised 
byte(s)
==10502==    at 0x3C86AE1973: __sendto_nocancel (in /lib64/libc-2.12.2.so)
==10502==    by 0x4C0BBAB: mnl_socket_sendto (socket.c:183)
==10502==    by 0x40CAFA: ipset_mnl_query (mnl.c:95)
==10502==    by 0x40AE28: build_send_private_msg (session.c:1567)
==10502==    by 0x40B897: ipset_cmd (session.c:1843)
==10502==    by 0x40C0FB: create_type_get (types.c:239)
==10502==    by 0x40C5D1: ipset_type_get (types.c:382)
==10502==    by 0x4065BA: ipset_parse_typename (parse.c:1389)
==10502==    by 0x40306E: parse_commandline (ipset.c:618)
==10502==    by 0x402204: restore (ipset.c:197)
==10502==    by 0x4031DF: parse_commandline (ipset.c:667)
==10502==    by 0x403485: main (ipset.c:742)
==10502==  Address 0x7fefffe09 is on thread 1's stack
==10502==  Uninitialised value was created by a stack allocation
==10502==    at 0x40ABE7: build_send_private_msg (session.c:1526)
==10502==
--10502-- REDIR: 0x3c86a7fcf0 (__GI_strncmp) redirected to 0x4a06800 
(__GI_strncmp)
--10502-- REDIR: 0x3c86a83490 (__GI_stpcpy) redirected to 0x4a072d0 
(__GI_stpcpy)
--10502-- REDIR: 0x3c86a82e50 (mempcpy) redirected to 0x4a07690 (mempcpy)
--10502-- REDIR: 0x3c86a84bb0 (strchrnul) redirected to 0x4a07620 
(strchrnul)
--10502-- REDIR: 0x3c86a7de00 (strcat) redirected to 0x4a05fb0 (strcat)
ipset v6.5: Error in line 1: Kernel error received: Operation not permitted
--10502-- REDIR: 0x3c86a7a2c0 (free) redirected to 0x4a04ceb (free)
==10502==
==10502== HEAP SUMMARY:
==10502==     in use at exit: 0 bytes in 0 blocks
==10502==   total heap usage: 4 allocs, 4 frees, 13,744 bytes allocated
==10502==
==10502== All heap blocks were freed -- no leaks are possible
==10502==
==10502== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)
==10502==
==10502== 1 errors in context 1 of 1:
==10502== Syscall param socketcall.sendto(msg) points to uninitialised 
byte(s)
==10502==    at 0x3C86AE1973: __sendto_nocancel (in /lib64/libc-2.12.2.so)
==10502==    by 0x4C0BBAB: mnl_socket_sendto (socket.c:183)
==10502==    by 0x40CAFA: ipset_mnl_query (mnl.c:95)
==10502==    by 0x40AE28: build_send_private_msg (session.c:1567)
==10502==    by 0x40B897: ipset_cmd (session.c:1843)
==10502==    by 0x40C0FB: create_type_get (types.c:239)
==10502==    by 0x40C5D1: ipset_type_get (types.c:382)
==10502==    by 0x4065BA: ipset_parse_typename (parse.c:1389)
==10502==    by 0x40306E: parse_commandline (ipset.c:618)
==10502==    by 0x402204: restore (ipset.c:197)
==10502==    by 0x4031DF: parse_commandline (ipset.c:667)
==10502==    by 0x403485: main (ipset.c:742)
==10502==  Address 0x7fefffe09 is on thread 1's stack
==10502==  Uninitialised value was created by a stack allocation
==10502==    at 0x40ABE7: build_send_private_msg (session.c:1526)
==10502==
--10502--
--10502-- used_suppression:      4 dl-hack3-cond-1
--10502-- used_suppression:      2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a
==10502==
==10502== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)

Quite revealing this!

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Quite revealing this!
Of course I have to remember to run this as root, so here it is (again):

[root@test1 src]# LD_LIBRARY_PATH=/home/me/src/libmnl-1.0.1/src/.libs 
libtool --mode=execute valgrind --track-origins=yes -v 
/home/me/src/ipset-6.5/src/ipset restore < ipset-6.5-segfault.ips
==10624== Memcheck, a memory error detector
==10624== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==10624== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==10624== Command: /home/me/src/ipset-6.5/src/ipset restore
==10624==
--10624-- Valgrind options:
--10624--    --track-origins=yes
--10624--    -v
--10624-- Contents of /proc/version:
--10624--   Linux version 2.6.35.13-91.fc13.x86_64 (me@test1.my.net) 
(gcc version 4.4.5 20101112 (Red Hat 4.4.5-2) (GCC) ) #1 SMP Sat May 21 
02:33:34 BST 2011
--10624-- Arch and hwcaps: AMD64, amd64-sse3-cx16
--10624-- Page sizes: currently 4096, max supported 4096
--10624-- Valgrind library directory: /usr/lib64/valgrind
--10624-- Reading syms from /home/me/src/ipset-6.5/src/ipset (0x400000)
--10624-- Reading syms from /usr/lib64/valgrind/memcheck-amd64-linux 
(0x38000000)
--10624--    object doesn't have a dynamic symbol table
--10624-- Reading syms from /lib64/ld-2.12.2.so (0x3c86600000)
--10624-- Reading suppressions file: /usr/lib64/valgrind/default.supp
--10624-- REDIR: 0x3c86617350 (strlen) redirected to 0x3803f6f7 
(vgPlain_amd64_linux_REDIR_FOR_strlen)
--10624-- Reading syms from 
/usr/lib64/valgrind/vgpreload_core-amd64-linux.so (0x4801000)
--10624-- Reading syms from 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so (0x4a02000)
==10624== WARNING: new redirection conflicts with existing -- ignoring it
--10624--     new: 0x3c86617350 (strlen              ) R-> 0x04a06340 strlen
--10624-- REDIR: 0x3c866171c0 (index) redirected to 0x4a05f80 (index)
--10624-- REDIR: 0x3c86617240 (strcmp) redirected to 0x4a06900 (strcmp)
--10624-- Reading syms from 
/home/me/src/libmnl-1.0.1/src/.libs/libmnl.so.0.0.1 (0x4c0a000)
--10624-- Reading syms from /lib64/libc-2.12.2.so (0x3c86a00000)
--10624-- REDIR: 0x3c86a815a0 (__GI_strrchr) redirected to 0x4a05e00 
(__GI_strrchr)
--10624-- REDIR: 0x3c86a7fab0 (strlen) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
==10624== WARNING: new redirection conflicts with existing -- ignoring it
--10624--     new: 0x3c86a7fae0 (__GI_strlen         ) R-> 0x04a062e0 strlen
--10624-- REDIR: 0x3c86a7fae0 (__GI_strlen) redirected to 0x4a06300 
(__GI_strlen)
--10624-- REDIR: 0x3c86a7e070 (strcmp) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
--10624-- REDIR: 0x3c86b26cc0 (__strcmp_ssse3) redirected to 0x4a06860 
(strcmp)
--10624-- REDIR: 0x3c86a79180 (calloc) redirected to 0x4a043d3 (calloc)
--10624-- REDIR: 0x3c86a7fcb0 (strncmp) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
--10624-- REDIR: 0x3c86b27f20 (__strncmp_ssse3) redirected to 0x4a067a0 
(strncmp)
--10624-- REDIR: 0x3c86a81cb0 (memchr) redirected to 0x4a069a0 (memchr)
--10624-- REDIR: 0x3c86a83730 (memcpy) redirected to 0x4a069d0 (memcpy)
--10624-- REDIR: 0x3c86a81540 (strncpy) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
--10624-- REDIR: 0x3c86b24140 (???) redirected to 0x4a06500 (strncpy)
--10624-- REDIR: 0xffffffffff600400 (???) redirected to 0x3803f6ed 
(vgPlain_amd64_linux_REDIR_FOR_vtime)
--10624-- REDIR: 0x3c86a82320 (memset) redirected to 0x4a07540 (memset)
==10624== Syscall param socketcall.sendto(msg) points to uninitialised 
byte(s)
==10624==    at 0x3C86AE1973: __sendto_nocancel (in /lib64/libc-2.12.2.so)
==10624==    by 0x4C0BBAB: mnl_socket_sendto (socket.c:183)
==10624==    by 0x40CAFA: ipset_mnl_query (mnl.c:95)
==10624==    by 0x40AE28: build_send_private_msg (session.c:1567)
==10624==    by 0x40B897: ipset_cmd (session.c:1843)
==10624==    by 0x40C0FB: create_type_get (types.c:239)
==10624==    by 0x40C5D1: ipset_type_get (types.c:382)
==10624==    by 0x4065BA: ipset_parse_typename (parse.c:1389)
==10624==    by 0x40306E: parse_commandline (ipset.c:618)
==10624==    by 0x402204: restore (ipset.c:197)
==10624==    by 0x4031DF: parse_commandline (ipset.c:667)
==10624==    by 0x403485: main (ipset.c:742)
==10624==  Address 0x7ff000199 is on thread 1's stack
==10624==  Uninitialised value was created by a stack allocation
==10624==    at 0x40ABE7: build_send_private_msg (session.c:1526)
==10624==
--10624-- REDIR: 0x3c86a79540 (malloc) redirected to 0x4a050db (malloc)
--10624-- REDIR: 0x3c86a7dfc0 (index) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
==10624== WARNING: new redirection conflicts with existing -- ignoring it
--10624--     new: 0x3c86a7dff0 (__GI_strchr         ) R-> 0x04a05e90 index
--10624-- REDIR: 0x3c86a7dff0 (__GI_strchr) redirected to 0x4a05ec0 
(__GI_strchr)
--10624-- REDIR: 0x3c86a7a2c0 (free) redirected to 0x4a04ceb (free)
--10624-- REDIR: 0x3c86a7f500 (strcpy) redirected to 0x480155c 
(_vgnU_ifunc_wrapper)
--10624-- REDIR: 0x3c86b23720 (???) redirected to 0x4a06360 (strcpy)
--10624-- REDIR: 0x3c86a84bb0 (strchrnul) redirected to 0x4a07620 
(strchrnul)
--10624-- REDIR: 0x3c86a82e50 (mempcpy) redirected to 0x4a07690 (mempcpy)
--10624-- REDIR: 0x3c86a7de00 (strcat) redirected to 0x4a05fb0 (strcat)
ipset v6.5: Error in line 3: Internal protocol error
==10624==
==10624== HEAP SUMMARY:
==10624==     in use at exit: 0 bytes in 0 blocks
==10624==   total heap usage: 11 allocs, 11 frees, 13,910 bytes allocated
==10624==
==10624== All heap blocks were freed -- no leaks are possible
==10624==
==10624== ERROR SUMMARY: 3 errors from 1 contexts (suppressed: 6 from 6)
==10624==
==10624== 3 errors in context 1 of 1:
==10624== Syscall param socketcall.sendto(msg) points to uninitialised 
byte(s)
==10624==    at 0x3C86AE1973: __sendto_nocancel (in /lib64/libc-2.12.2.so)
==10624==    by 0x4C0BBAB: mnl_socket_sendto (socket.c:183)
==10624==    by 0x40CAFA: ipset_mnl_query (mnl.c:95)
==10624==    by 0x40AE28: build_send_private_msg (session.c:1567)
==10624==    by 0x40B897: ipset_cmd (session.c:1843)
==10624==    by 0x40C0FB: create_type_get (types.c:239)
==10624==    by 0x40C5D1: ipset_type_get (types.c:382)
==10624==    by 0x4065BA: ipset_parse_typename (parse.c:1389)
==10624==    by 0x40306E: parse_commandline (ipset.c:618)
==10624==    by 0x402204: restore (ipset.c:197)
==10624==    by 0x4031DF: parse_commandline (ipset.c:667)
==10624==    by 0x403485: main (ipset.c:742)
==10624==  Address 0x7ff000199 is on thread 1's stack
==10624==  Uninitialised value was created by a stack allocation
==10624==    at 0x40ABE7: build_send_private_msg (session.c:1526)
==10624==
--10624--
--10624-- used_suppression:      4 dl-hack3-cond-1
--10624-- used_suppression:      2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a
==10624==
==10624== ERROR SUMMARY: 3 errors from 1 contexts (suppressed: 6 from 6)

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Your first set named 'test_ips' consisting of 11670 ranges can only be 
> converted to 15613 networks, i.e. members. Not all ranges can be expressed 
> with a single network.
>   
I have a suggestion then: Is it possible to include the number of 
members registered in the set in the set headers - something like 
"Members: xxxxxx"? That way I won't be guessing how much members I have 
in a set (particularly if members of said set run into many thousands) 
and don't rely (as I did previously) on my text files to do "estimates", 
which, as we've seen, could be wrong and misleading and I would be able 
to view this with the "-terse" option.

Re: [ANNOUNCE] ipset 6.5 released

[Jan Engelhardt]

On Sunday 2011-05-22 01:08, Jozsef Kadlecsik wrote:
>
>==2229== Syscall param socketcall.sendto(msg) points to uninitialised byte(s)
>==2229==    at 0x5118183: sendto (in /lib/libc-2.9.so)
>==2229==    by 0x4E2DBDB: mnl_socket_sendto (socket.c:183)
>==2229==    by 0x40CD06: ipset_mnl_query (mnl.c:95)
>==2229==    by 0x40B032: build_send_private_msg (session.c:1567)
>==2229==    by 0x40BAA1: ipset_cmd (session.c:1843)
>==2229==    by 0x40C307: create_type_get (types.c:239)
>==2229==    by 0x40C7DD: ipset_type_get (types.c:382)
>==2229==    by 0x4067CA: ipset_parse_typename (parse.c:1389)
>==2229==    by 0x40327E: parse_commandline (ipset.c:618)
>==2229==    by 0x402414: restore (ipset.c:197)
>==2229==    by 0x4033EF: parse_commandline (ipset.c:667)
>==2229==    by 0x403695: main (ipset.c:742)
>==2229==  Address 0x7ff000269 is on thread 1's stack
>ipset v6.5: Error in line 11191: Hash is full, cannot add more elements
>==2229== 
>==2229== ERROR SUMMARY: 2 errors from 1 contexts (suppressed: 8 from 1)
>==2229== malloc/free: in use at exit: 0 bytes in 0 blocks.
>==2229== malloc/free: 67,277 allocs, 67,277 frees, 2,385,155 bytes allocated.
>==2229== For counts of detected errors, rerun with: -v
>==2229== Use --track-origins=yes to see where uninitialised values come from
>==2229== All heap blocks were freed -- no leaks are possible.
>
>The uninitialized bytes are OK and there is no memory leak or corruption.

Uninitialized bytes are usually never OK when they are read upon.
There is a series on LWN to elaborate on that..

Re: [ANNOUNCE] ipset 6.5 released

[Jozsef Kadlecsik]

On Sun, 22 May 2011, Mr Dash Four wrote:

> > If you want me to be able to reproduce and fix the issue, please send me the
> > full data you used. I assumed the restore files you sent me contained the
> > case where the segfaults popped up. Apparently it's not, because there's no
> > such commands in them.
> >   
> Try this and let me know how far do you get:
> 
> n priviledged-ports bitmap:port range 1-1023 timeout 0
> a priviledged-ports 1-1023
> n test-net bitmap:ip range 10.1.1.172/29 timeout 0
> a test-net 10.1.1.172/29
> n test-ports bitmap:port range 12770-19999 timeout 0
> a test-ports 12770-12784
> a test-ports 19999
> n test-port bitmap:port range 29950-29950 timeout 0
> a test-port 29950
> COMMIT

Thanks, that helped a lot! The ipset binary doesn't handle different sets 
in a single restore file. I'll fix it in the next days.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: [ANNOUNCE] ipset 6.5 released

[Mr Dash Four]

> Thanks, that helped a lot! The ipset binary doesn't handle different sets 
> in a single restore file. I'll fix it in the next days.
>   
Interesting! The file I had with the 3 hash:net sets in it seems to be 
restored OK without any problems after applying the patch I've got from 
you yesterday (the sets I enclosed in the post you quoted were from a 
different file though). The problematic statements seem to be bitmap 
sets (bitmap:ip as well as bitmap:port)- it also looked to be choking on 
IP addresses specified as cidr (i.e. x.x.x.x/x). Anyway, let me know if 
you need anything else from me.