Netlink limitations

Netlink limitations

[Jan Engelhardt]

Hi,


we mentioned it only briefly at the Netfilter workshop a few weeks ago, 
but as I am trying to figure out how to use Netlink in Xtables, 
Netlink's limitations really start ruining my day.

The well-known issue is that NL messages the kernel is supposed to 
receive have a max size of 64K, due to nlmsghdr's use of uint16_t. This 
is very problematic because attributes can easily amass more than 64K. 
Think of a chain full of rules, represented by a top-level attribute 
that nests attributes. The problem is bidirectional, a table 
dump has the same problem.

A further problem seems to be that the kernel does not seem to have 
support for receiving NLM_F_MULTI messages, so even assuming chains were 
just 40K, one cannot atomically replace an entire table with 2 chains of 
40K each. Trying to slap transaction support on _top_ of netlink is not 
going to work with the current implementation, because there is no 
notification of when the socket is closed before a NLMSG_DONE has been 
sent.

What I would also like is streaming support, i.e. that I can tag an 
attribute container (one that has nested attrs) with .len = -1 to define 
that the end of the container is given not by .len, but by a stop 
marker.

Hacks like nfnetlink or genetlink also seem unnecessary to me, and the 
limit of MAX_LINKS=32 most likely just stems from nl_table being an 
array that is not very sparse.

Perhaps it is time to replace Netlink by something new?
Trying to elicit some opinions.


Jan

Re: Netlink limitations

[Patrick McHardy]

On 07.11.2010 17:44, Jan Engelhardt wrote:
> we mentioned it only briefly at the Netfilter workshop a few weeks ago, 
> but as I am trying to figure out how to use Netlink in Xtables, 
> Netlink's limitations really start ruining my day.
> 
> The well-known issue is that NL messages the kernel is supposed to 
> receive have a max size of 64K, due to nlmsghdr's use of uint16_t. This 
> is very problematic because attributes can easily amass more than 64K. 
> Think of a chain full of rules, represented by a top-level attribute 
> that nests attributes. The problem is bidirectional, a table 
> dump has the same problem.

Messages are not limited to 64k, individual attributes are. Holger
started working on a nlattr32, which uses 32 bit for the length
value.

> A further problem seems to be that the kernel does not seem to have 
> support for receiving NLM_F_MULTI messages, so even assuming chains were 
> just 40K, one cannot atomically replace an entire table with 2 chains of 
> 40K each. Trying to slap transaction support on _top_ of netlink is not 
> going to work with the current implementation, because there is no 
> notification of when the socket is closed before a NLMSG_DONE has been 
> sent.

There is, search for NETLINK_URELEASE in af_netlink.c. With 32 bit
attribute lengths this should not be needed anymore however.

> What I would also like is streaming support, i.e. that I can tag an 
> attribute container (one that has nested attrs) with .len = -1 to define 
> that the end of the container is given not by .len, but by a stop 
> marker.

That's somewhat similar to the nlattr32 idea, but a length of 0 makes
more sense since that's currently not used. In that case the length
would be read from a second length field which has 32 bits.

Re: Netlink limitations

[Thomas Graf]

On Sun, Nov 07, 2010 at 06:17:43PM +0100, Patrick McHardy wrote:
> On 07.11.2010 17:44, Jan Engelhardt wrote:
> > we mentioned it only briefly at the Netfilter workshop a few weeks ago, 
> > but as I am trying to figure out how to use Netlink in Xtables, 
> > Netlink's limitations really start ruining my day.
> > 
> > The well-known issue is that NL messages the kernel is supposed to 
> > receive have a max size of 64K, due to nlmsghdr's use of uint16_t. This 
> > is very problematic because attributes can easily amass more than 64K. 
> > Think of a chain full of rules, represented by a top-level attribute 
> > that nests attributes. The problem is bidirectional, a table 
> > dump has the same problem.
> 
> Messages are not limited to 64k, individual attributes are. Holger
> started working on a nlattr32, which uses 32 bit for the length
> value.

Also, it is not required to pack everything in attributes. Your protocol
may specify that the whole message payload consists of chained attributes.
Alternatively you may as well split your attribut chain and dump them
as several messages.

If a large attribute container for nested attribtues is required, I agree
that nlattr32 is the way to go.

> That's somewhat similar to the nlattr32 idea, but a length of 0 makes
> more sense since that's currently not used. In that case the length
> would be read from a second length field which has 32 bits.

I agree. This makes perfect sense. I was just thinking wheter we want
to encode more information into the attribute header if we extend it
anyways.

Re: Netlink limitations

[Jan Engelhardt]

On Monday 2010-11-08 16:16, Thomas Graf wrote:
>> 
>> Messages are not limited to 64k, individual attributes are. Holger
>> started working on a nlattr32, which uses 32 bit for the length
>> value.
>
>Also, it is not required to pack everything in attributes. Your protocol
>may specify that the whole message payload consists of chained attributes.
>Alternatively you may as well split your attribut chain and dump them
>as several messages.

Yeah with NETLINK_URELEASE that seems the way to go. However, what are
compelling arguments to use Netlink over other forms of bidirectional
communication? (To play devils advocate, one could use nlattr32/TLVs
over ioctl too.)

Re: Netlink limitations

[Pablo Neira Ayuso]

On 08/11/10 20:21, Jan Engelhardt wrote:
> On Monday 2010-11-08 16:16, Thomas Graf wrote:
>>>
>>> Messages are not limited to 64k, individual attributes are. Holger
>>> started working on a nlattr32, which uses 32 bit for the length
>>> value.
>>
>> Also, it is not required to pack everything in attributes. Your protocol
>> may specify that the whole message payload consists of chained attributes.
>> Alternatively you may as well split your attribut chain and dump them
>> as several messages.
> 
> Yeah with NETLINK_URELEASE that seems the way to go. However, what are
> compelling arguments to use Netlink over other forms of bidirectional
> communication? (To play devils advocate, one could use nlattr32/TLVs
> over ioctl too.)

Netlink also provides an event-based notification infrastructure. Of
course, you can implement that upon a new socket family that supports
your new ioctls operations taking things in TLV format.

However, I guess that the whole thing will start looking like netlink
quite a lot in the end ;-).

Re: Netlink limitations

[Patrick McHardy]

Am 08.11.2010 16:16, schrieb Thomas Graf:
> On Sun, Nov 07, 2010 at 06:17:43PM +0100, Patrick McHardy wrote:
>> On 07.11.2010 17:44, Jan Engelhardt wrote:
>>> ...
> 
>> That's somewhat similar to the nlattr32 idea, but a length of 0 makes
>> more sense since that's currently not used. In that case the length
>> would be read from a second length field which has 32 bits.
> 
> I agree. This makes perfect sense. I was just thinking wheter we want
> to encode more information into the attribute header if we extend it
> anyways.

What kind of additional information are you thinking about?

Re: Netlink limitations

[Jan Engelhardt]

On Monday 2010-11-08 16:16, Thomas Graf wrote:
>
>Also, it is not required to pack everything in attributes. Your protocol
>may specify that the whole message payload consists of chained attributes.

Yeah but that does not work well with nla_parse and nfnetlink.c
unserializing the byte stream into a tb[] array.

Does it hurt to not use nfnetlink? The only issue I see is that
af_netlink.c's nl_table is an array, when it should preferably be a
hash or tree if we are going to add more subsystems.

Re: Netlink limitations

[Jan Engelhardt]

On Sunday 2010-11-07 18:17, Patrick McHardy wrote:
>On 07.11.2010 17:44, Jan Engelhardt wrote:
>> we mentioned it only briefly at the Netfilter workshop a few weeks ago, 
>> but as I am trying to figure out how to use Netlink in Xtables, 
>> Netlink's limitations really start ruining my day.
>> 
>> The well-known issue is that NL messages[sic] the kernel is supposed to 
>> receive have a max size of 64K, due to nlmsghdr's use of uint16_t. This 
>> is very problematic because attributes can easily amass more than 64K. 
>> Think of a chain full of rules, represented by a top-level attribute 
>> that nests attributes. The problem is bidirectional, a table 
>> dump has the same problem.
>
>Messages are not limited to 64k, individual attributes are. Holger
>started working on a nlattr32, which uses 32 bit for the length
>value.

Does he have a format specification available?

Re: Netlink limitations

[Patrick McHardy]

Am 09.11.2010 13:10, schrieb Jan Engelhardt:
> 
> On Sunday 2010-11-07 18:17, Patrick McHardy wrote:
>> On 07.11.2010 17:44, Jan Engelhardt wrote:
>>> we mentioned it only briefly at the Netfilter workshop a few weeks ago, 
>>> but as I am trying to figure out how to use Netlink in Xtables, 
>>> Netlink's limitations really start ruining my day.
>>>
>>> The well-known issue is that NL messages[sic] the kernel is supposed to 
>>> receive have a max size of 64K, due to nlmsghdr's use of uint16_t. This 
>>> is very problematic because attributes can easily amass more than 64K. 
>>> Think of a chain full of rules, represented by a top-level attribute 
>>> that nests attributes. The problem is bidirectional, a table 
>>> dump has the same problem.
>>
>> Messages are not limited to 64k, individual attributes are. Holger
>> started working on a nlattr32, which uses 32 bit for the length
>> value.
> 
> Does he have a format specification available?

As I said, the basic idea is to use a length value of zero to indicate
that the length should be read from a second length member. Basically:

struct nlattr32 {
	__u16 nla_len;
	__u16 nla_type;
	__u32 nla_len2;
};

Re: Netlink limitations

[Thomas Graf]

On Tue, Nov 09, 2010 at 10:27:34AM +0100, Patrick McHardy wrote:
> Am 08.11.2010 16:16, schrieb Thomas Graf:
> > On Sun, Nov 07, 2010 at 06:17:43PM +0100, Patrick McHardy wrote:
> >> On 07.11.2010 17:44, Jan Engelhardt wrote:
> >>> ...
> > 
> >> That's somewhat similar to the nlattr32 idea, but a length of 0 makes
> >> more sense since that's currently not used. In that case the length
> >> would be read from a second length field which has 32 bits.
> > 
> > I agree. This makes perfect sense. I was just thinking wheter we want
> > to encode more information into the attribute header if we extend it
> > anyways.
> 
> What kind of additional information are you thinking about?

We have tried to come up with ways of forwarding netlink messages to
other machines several times. It always failed due to the fact that
protocols encode attributes/data differently without having the
ability to specify the encoding. E.g. we have nfnetlink encoding
everything in network byte order, most others encode it in host byte
order. Therefore all parsers must be aware of all protocols if they
want to forward/do something with the data. A flag would help there.

I haven't given up on the idea of self describing netlink protocols
yet. For example we could encode the attribute type
(i8|u16|u32|u16|string) in additional to the existing nested attribute
flag.

Given this additional meta data to each attribute and given we limit
ourselves to only using strict netlink attribtes going forward, i.e.
we no longer encode whole structs in attributes we'd be given netlink
protocols which can be forwarded, saved, audited while still being
fairly efficient.

Re: Netlink limitations

[Jan Engelhardt]

On Tuesday 2010-11-09 15:49, Thomas Graf wrote:
>
>We have tried to come up with ways of forwarding netlink messages to
>other machines several times. It always failed due to the fact that
>protocols encode attributes/data differently without having the
>ability to specify the encoding.

What's more, there is no way to specify a remote host in sockaddr_nl
right now, so all communication is necessarily being local - that is,
unless you add a hidden forwarder in kernel space that transparently
tunnels it into IPv6 or something.

>I haven't given up on the idea of self describing netlink protocols
>yet. For example we could encode the attribute type
>(i8|u16|u32|u16|string) in additional to the existing nested attribute
>flag.

I do not believe that encoding the attribute type into the protocol
itself is going to be such a big win. You still need a local
authoritative database (struct nla_policy[] or some representation of
it, nevermind I'm thinking "XML-DTD"-like) to do the verification
against because some NL messages may be purposely forged. If you have
an nlattr that says it is a string, how do you know that it is in
fact a string rather than a blob that happens to have a trailing \0.

Re: Netlink limitations

[Thomas Graf]

On Tue, Nov 09, 2010 at 09:20:09PM +0100, Jan Engelhardt wrote:
> What's more, there is no way to specify a remote host in sockaddr_nl
> right now, so all communication is necessarily being local - that is,
> unless you add a hidden forwarder in kernel space that transparently
> tunnels it into IPv6 or something.

That's fine. I don't expect the kernel to send netlink message to
other machines directly but rather have a userspace daemon handle
this. 

> I do not believe that encoding the attribute type into the protocol
> itself is going to be such a big win. You still need a local
> authoritative database (struct nla_policy[] or some representation of
> it, nevermind I'm thinking "XML-DTD"-like) to do the verification
> against because some NL messages may be purposely forged. If you have
> an nlattr that says it is a string, how do you know that it is in
> fact a string rather than a blob that happens to have a trailing \0.

True, we will never be able to verify the contents of attributes but
what we can do is give the sender the ability to specify what type of
attribute he was meant to send. This can be a big advantage as it
limits the possibliy of misinterpreting messages which may have been
corrupted because we can match the expected attribute type against
the attribute type supplied by the sender. Of course this doesn't
protect against forged messages at all, we will never be able to do
that.

The addition won't be a revolution but it the increased header size,
8 vs. 12 bytes isn't a big deal and gives us some additional room to
work with in the future.

struct nlattr_ext {
	u16	oldlen;		/* 0 */
	u16	kind;		/* TCA_* */
	u8	type;		/* NLA_U32 */
	u8	flags;		/* NLA_F_* */
	u16	reserved;
	u32	length;
};

There has been more than one debate whether to share nla_policy between
kernel and userspace. There is nothing which prevents people from doing
so. But typically the semantics between kernel->userspace and vice versa
are slightly different and require a different policy to be applied.

Re: Netlink limitations

[Jan Engelhardt]

On Tuesday 2010-11-09 22:40, Thomas Graf wrote:
>The addition won't be a revolution but it the increased header size,
>8 vs. 12 bytes isn't a big deal and gives us some additional room to
>work with in the future.
>
>struct nlattr_ext {
>	u16	oldlen;		/* 0 */
>	u16	kind;		/* TCA_* */
>	u8	type;		/* NLA_U32 */
>	u8	flags;		/* NLA_F_* */
>	u16	reserved;
>	u32	length;
>};

And while we're discussing this, surely there are no objections
to bumping NLA_ALIGN to 8 at the same time..

Re: Netlink limitations

[Thomas Graf]

On Tue, Nov 09, 2010 at 11:02:30PM +0100, Jan Engelhardt wrote:
> 
> On Tuesday 2010-11-09 22:40, Thomas Graf wrote:
> >The addition won't be a revolution but it the increased header size,
> >8 vs. 12 bytes isn't a big deal and gives us some additional room to
> >work with in the future.
> >
> >struct nlattr_ext {
> >	u16	oldlen;		/* 0 */
> >	u16	kind;		/* TCA_* */
> >	u8	type;		/* NLA_U32 */
> >	u8	flags;		/* NLA_F_* */
> >	u16	reserved;
> >	u32	length;
> >};
> 
> And while we're discussing this, surely there are no objections
> to bumping NLA_ALIGN to 8 at the same time..

We can't do that. That would break _everything_.

Re: Netlink limitations

[Jan Engelhardt]

On Wednesday 2010-11-10 00:35, Thomas Graf wrote:

>On Tue, Nov 09, 2010 at 11:02:30PM +0100, Jan Engelhardt wrote:
>> 
>> On Tuesday 2010-11-09 22:40, Thomas Graf wrote:
>> >The addition won't be a revolution but it the increased header size,
>> >8 vs. 12 bytes isn't a big deal and gives us some additional room to
>> >work with in the future.
>> >
>> >struct nlattr_ext {
>> >	u16	oldlen;		/* 0 */
>> >	u16	kind;		/* TCA_* */
>> >	u8	type;		/* NLA_U32 */
>> >	u8	flags;		/* NLA_F_* */
>> >	u16	reserved;
>> >	u32	length;
>> >};
>> 
>> And while we're discussing this, surely there are no objections
>> to bumping NLA_ALIGN to 8 at the same time..
>
>We can't do that. That would break _everything_.

Using nlattr_ext also breaks _something_, unless it's only used for new 
stuff. Similarly, a new NLA_EXT_ALIGN could be used with just those that 
also start using nlattr_ext.

Re: Netlink limitations

[Thomas Graf]

On Wed, Nov 10, 2010 at 12:42:25AM +0100, Jan Engelhardt wrote:
> On Wednesday 2010-11-10 00:35, Thomas Graf wrote:
> >On Tue, Nov 09, 2010 at 11:02:30PM +0100, Jan Engelhardt wrote:
> >> And while we're discussing this, surely there are no objections
> >> to bumping NLA_ALIGN to 8 at the same time..
> >
> >We can't do that. That would break _everything_.
> 
> Using nlattr_ext also breaks _something_, unless it's only used for new 
> stuff. Similarly, a new NLA_EXT_ALIGN could be used with just those that 
> also start using nlattr_ext.

If you want to change alignment for new protocols that's fine but you won't
be able to use the existing attribute API on either side.

nlattr_ext2 or nlattr32 could be used in existing protocols if we used a special
attribute type (f.e. 0xffff) instead of nla_len == 0 to identify them and as long
as the attribute size does not exceed 64K as obviously no older parser would be
able to skip over such attributes correctly.

So yes, large attributes would only be permitted in new protocols which are
guaranteed to have a capable parser but we would at least not have to duplicate
the API but just slightly extend it.