* [Nftables RFC] High level library proposal @ 2013-04-17 13:41 Tomasz Bursztyka 2013-04-17 13:52 ` Victor Julien 2013-04-19 10:05 ` Pablo Neira Ayuso 0 siblings, 2 replies; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-17 13:41 UTC (permalink / raw) To: Netfilter Development Mailing list Cc: Patrick McHardy, Pablo Neira Ayuso, Eric Leblond, Julien Vehent NFTABLES - High level library proposal ====================================== The goal is to enable applications to get an easy and flexible access to nftables netlink's API. We have to consider a lot of different use-case, from the application which want to manage the whole rule set to the one which will quickly do a one-shot access. I tried to keep the API simple so it will be almost as simple as what current applications are doing with iptables plus the extra nice stuff nftables bring. (notifications, transactions mostly...). All complex work should be hidden to the developer. Thus, it will take a bit of time to get it implemented.w Table/chain/rule/set specifications will follow nft format. The library will not provide backward compatibility to iptables format. The tools which access existing xtables do it through the iptables tool, so if they want to access nftables with backward compatible rule format it will use iptables-nftables tool instead. So no need to support iptables format in this library. Using this library presented here will be already quite a task, so I guess moving towards using nft format as well will not be the hard part anyway. I am thinking of taking original nft tool's lexer/parser for the statements. It's nice, it works, no need to reinvent the wheel. The change will be on creating libnftables objects instead. To validate the event and nl driver approach I did a PoC which proved it. I could create a table and chain, list those... witheither libmnl or libnl, and glib as the event loop, it just works so it opens compatibility against any netlinkaccess library or event loop (libevent, efl...) Note: It's just a proposal, so the function names/signature is about to change most surely. But the basic idea is there. API: === int nft_init(int flags, struct nft_event_driver *event_driver, struct nft_nl_driver nl_driver): -> initiate the library. Flags: NFT_FLAG_NL_SYNC (default): using libmnl (already used by libnftables, so no extra dependancy) event_driver and nl_driver are forgotten. NFT_FLAG_NL_ASYNC: Same as previous but a event_driver is at least mandatory. And a nl_driver can be provided to support libnl, others... I am thinking of those as well: NFT_FLAG_FOLLOW_LOCAL (default): will notice about the locally manipulated rules events. NFT_FLAG_FOLLOW_GLOBAL: will keep notice about the whole rule set event. Return: 0 on success, negative error value instead. void nft_exit(void): -> exit the library. Basic operations: ================ Signature of callback has to be designed further. Event type, handle, user data, ... int nft_set_notification_callback(nft_notification_callback_f): -> notify on nftables event (NFT_FLAG_FOLLOW_* applies) int nft_execute_statement(nft_statement_callback_f, void *user_data, char *statement_format, ...): -> execute statement. Execution flow depends on NFT_FLAG_NL_* of course. This could be compared to what is done currently with iptables commands thrown through a sub-process. But here, no sub-process, notifications, ... Context based: ============= When nft_execute_statement() is nice for one-shot call and forget, some applications might want to follow precisely what's happening in a long-run on nftables rule-set either in general or depending on specific context which can be plural in same application. One might need to temporarilyset a certain number of rules dedicated to a situation. This part of the API will help on that. From internal execution point of view, it will be transaction based. Callback signature will probably follow the same as previous ones, with slight difference like adding the context pointer to it. struct nft_ctx *nft_ctx_new(nft_context_callback_f) -> create a new context. int nft_ctx_list(struct nft_ctx *ctx, FILE *out) -> list the context's owned statements int nft_ctx_enable(struct nft_ctx *ctx) -> apply statements previously executed int nft_ctx_disable(struct nft_ctx *ctx) -> delete statements previously executed void nft_ctx_free(struct nft_ctx *ctx) -> free the context (do not enable/disable anything here) int nft_ctx_execute_statement(struct nft_ctx *ctx, char *statement_format, ...) -> execute the statement related to given context. Listing: ======= int nft_setup_list_driver(int flags, struct nft_list_driver *driver): -> set the output format for listing the statements flags: NFT_LIST_DRIVER_XML (current default format in libnftables) NFT_LIST_DRIVER_JSON (that format idea comes from Julien Vehent) NFT_LIST_DRIVER_EXTERNAL -> driver should be provided (need small modifications in libnftables) int nft_list_all(FILE *out) We could have specific things like nft_list_by_handle()... not sure if it will be useful. Besides, we could have lower level helpers, like nft_table_new(char *table_name) or whatever comes. If relevant, only. If you have comments and/or ideas, please go ahead. I am anyway starting to implement itnowand I will in parallel create real tools to test it according to specific use cases. @Pablo: could you create a repository on netfilter.org's git? libnft? or whatever comes as better name if anybody has a nice proposition Br, Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-17 13:41 [Nftables RFC] High level library proposal Tomasz Bursztyka @ 2013-04-17 13:52 ` Victor Julien 2013-04-19 6:50 ` Tomasz Bursztyka 2013-04-19 10:05 ` Pablo Neira Ayuso 1 sibling, 1 reply; 20+ messages in thread From: Victor Julien @ 2013-04-17 13:52 UTC (permalink / raw) To: Tomasz Bursztyka Cc: Netfilter Development Mailing list, Patrick McHardy, Pablo Neira Ayuso, Eric Leblond, Julien Vehent On 04/17/2013 03:41 PM, Tomasz Bursztyka wrote: > NFTABLES - High level library proposal > ====================================== > > The goal is to enable applications to get an easy and flexible access to > nftables netlink's API. > > We have to consider a lot of different use-case, from the application which > want to manage the whole rule set to the one which will quickly do a > one-shot > access. > > I tried to keep the API simple so it will be almost as simple as what > current > applications are doing with iptables plus the extra nice stuff nftables > bring. > (notifications, transactions mostly...). All complex work should be > hidden to > the developer. Thus, it will take a bit of time to get it implemented.w > > Table/chain/rule/set specifications will follow nft format. The library > will > not provide backward compatibility to iptables format. The tools which > access > existing xtables do it through the iptables tool, so if they want to access > nftables with backward compatible rule format it will use iptables-nftables > tool instead. So no need to support iptables format in this library. > Using this library presented here will be already quite a task, so I guess > moving towards using nft format as well will not be the hard part anyway. > > I am thinking of taking original nft tool's lexer/parser for the > statements. > It's nice, it works, no need to reinvent the wheel. The change will be on > creating libnftables objects instead. > > To validate the event and nl driver approach I did a PoC which proved it. I > could create a table and chain, list those... witheither libmnl or > libnl, and > glib as the event loop, it just works so it opens compatibility against any > netlinkaccess library or event loop (libevent, efl...) > > Note: It's just a proposal, so the function names/signature is about to > change > most surely. But the basic idea is there. > > > API: > === > > int nft_init(int flags, struct nft_event_driver *event_driver, > struct nft_nl_driver nl_driver): > -> initiate the library. > > Flags: > NFT_FLAG_NL_SYNC (default): using libmnl (already used by libnftables, > so no extra dependancy) event_driver and nl_driver are forgotten. > NFT_FLAG_NL_ASYNC: Same as previous but a event_driver is at least > mandatory. > And a nl_driver can be provided to support libnl, others... > > I am thinking of those as well: > > NFT_FLAG_FOLLOW_LOCAL (default): will notice about the locally manipulated > rules events. > NFT_FLAG_FOLLOW_GLOBAL: will keep notice about the whole rule set event. > > Return: 0 on success, negative error value instead. > > void nft_exit(void): > -> exit the library. > > > Basic operations: > ================ > > Signature of callback has to be designed further. Event type, handle, > user data, ... > > int nft_set_notification_callback(nft_notification_callback_f): > -> notify on nftables event (NFT_FLAG_FOLLOW_* applies) > > int nft_execute_statement(nft_statement_callback_f, void *user_data, > char *statement_format, ...): > -> execute statement. Execution flow depends on NFT_FLAG_NL_* of course. > This could be compared to what is done currently with iptables commands > thrown through a sub-process. But here, no sub-process, notifications, ... > > > Context based: > ============= > > When nft_execute_statement() is nice for one-shot call and forget, some > applications might want to follow precisely what's happening in a > long-run on > nftables rule-set either in general or depending on specific context > which can > be plural in same application. One might need to temporarilyset a certain > number of rules dedicated to a situation. This part of the API will help on > that. > > From internal execution point of view, it will be transaction based. > > Callback signature will probably follow the same as previous ones, with > slight difference like adding the context pointer to it. > > struct nft_ctx *nft_ctx_new(nft_context_callback_f) > -> create a new context. > > int nft_ctx_list(struct nft_ctx *ctx, FILE *out) > -> list the context's owned statements > > int nft_ctx_enable(struct nft_ctx *ctx) > -> apply statements previously executed > > int nft_ctx_disable(struct nft_ctx *ctx) > -> delete statements previously executed > > void nft_ctx_free(struct nft_ctx *ctx) > -> free the context (do not enable/disable anything here) > > int nft_ctx_execute_statement(struct nft_ctx *ctx, > char *statement_format, ...) > -> execute the statement related to given context. > > > Listing: > ======= > > int nft_setup_list_driver(int flags, > struct nft_list_driver *driver): > -> set the output format for listing the statements > flags: NFT_LIST_DRIVER_XML (current default format in libnftables) > NFT_LIST_DRIVER_JSON (that format idea comes from Julien Vehent) > NFT_LIST_DRIVER_EXTERNAL -> driver should be provided (need small > modifications in libnftables) > > int nft_list_all(FILE *out) > > We could have specific things like nft_list_by_handle()... not sure if > it will > be useful. > > Besides, we could have lower level helpers, like nft_table_new(char > *table_name) or whatever comes. If relevant, only. > > If you have comments and/or ideas, please go ahead. I am anyway starting to > implement itnowand I will in parallel create real tools to test it > according > to specific use cases. > > > @Pablo: could you create a repository on netfilter.org's git? libnft? > > or whatever comes as better name if anybody has a nice proposition > > Not sure if it would fit the scope of this library, but as a frontend developer I would love to have easy access to some sort of "supported features" call. In Vuurmuur I currently parse /proc/net/ip_tables_names to see what tables are supported, /proc/net/ip_tables_matches to see what matches are supported, etc. This still isn't enough, because it won't tell me if the SNAT target will actually support the --random option, so I end up creating a lot of test rules at startup, just to figure this stuff out. Then there is also the case of a mismatch between kernel and userland. I remember one case where the Ubuntu kernel would support a module, but the shipped iptables wouldn't. Not sure if all of this is relevant to nftables and I don't have a proposed solution, but just wanted to bring it up for consideration. -- --------------------------------------------- Victor Julien http://www.inliniac.net/ PGP: http://www.inliniac.net/victorjulien.asc --------------------------------------------- ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-17 13:52 ` Victor Julien @ 2013-04-19 6:50 ` Tomasz Bursztyka 0 siblings, 0 replies; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-19 6:50 UTC (permalink / raw) To: Victor Julien Cc: Netfilter Development Mailing list, Patrick McHardy, Pablo Neira Ayuso, Eric Leblond, Julien Vehent Hi Victor, > Not sure if it would fit the scope of this library, but as a frontend > developer I would love to have easy access to some sort of "supported > features" call. > > In Vuurmuur I currently parse /proc/net/ip_tables_names to see what > tables are supported, /proc/net/ip_tables_matches to see what matches > are supported, etc. > > This still isn't enough, because it won't tell me if the SNAT target > will actually support the --random option, so I end up creating a lot of > test rules at startup, just to figure this stuff out. > > Then there is also the case of a mismatch between kernel and userland. I > remember one case where the Ubuntu kernel would support a module, but > the shipped iptables wouldn't. > > Not sure if all of this is relevant to nftables and I don't have a > proposed solution, but just wanted to bring it up for consideration. This is a good idea, since indeed not all features might be supported from one kernel configuration/version to another. However, nftables does not expose anything through proc-fs currently. And it does not tell anything about what are supported features anywhere, afaik. We should first think how to fix this from kernel side, for the library itself it should be trivial afterwards. There are issues like as long as modules are not loaded you don't know for instance which expressions are supported... Maybe kernel guys have good ideas how to fix this? Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-17 13:41 [Nftables RFC] High level library proposal Tomasz Bursztyka 2013-04-17 13:52 ` Victor Julien @ 2013-04-19 10:05 ` Pablo Neira Ayuso 2013-04-19 11:26 ` Tomasz Bursztyka 2013-04-22 20:05 ` Jesper Dangaard Brouer 1 sibling, 2 replies; 20+ messages in thread From: Pablo Neira Ayuso @ 2013-04-19 10:05 UTC (permalink / raw) To: Tomasz Bursztyka Cc: Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent Hi Tomasz, On Wed, Apr 17, 2013 at 04:41:24PM +0300, Tomasz Bursztyka wrote: > NFTABLES - High level library proposal > ====================================== > > The goal is to enable applications to get an easy and flexible access to > nftables netlink's API. > > We have to consider a lot of different use-case, from the application which > want to manage the whole rule set to the one which will quickly do a > one-shot > access. > > I tried to keep the API simple so it will be almost as simple as > what current applications are doing with iptables plus the extra > nice stuff nftables bring. (notifications, transactions mostly...). > All complex work should be hidden to the developer. Thus, it will > take a bit of time to get it implemented.w > > Table/chain/rule/set specifications will follow nft format. The > library will not provide backward compatibility to iptables format. > The tools which access existing xtables do it through the iptables > tool, so if they want to access nftables with backward compatible > rule format it will use iptables-nftables tool instead. So no need > to support iptables format in this library. Using this library > presented here will be already quite a task, so I guess moving > towards using nft format as well will not be the hard part anyway. > > I am thinking of taking original nft tool's lexer/parser for the statements. > It's nice, it works, no need to reinvent the wheel. The change will be on > creating libnftables objects instead. We can just export the abstract syntax tree functions to build the rules. I've also discussing with Patrick that some even higher level API, something simple for users, just like "match tcp port 22" would be good to have. > To validate the event and nl driver approach I did a PoC which > proved it. I could create a table and chain, list those... > witheither libmnl or libnl, and glib as the event loop, it just > works so it opens compatibility against any netlinkaccess library or > event loop (libevent, efl...) > > Note: It's just a proposal, so the function names/signature is about > to change most surely. But the basic idea is there. > > > API: > === > > int nft_init(int flags, struct nft_event_driver *event_driver, > struct nft_nl_driver nl_driver): > -> initiate the library. > > Flags: > NFT_FLAG_NL_SYNC (default): using libmnl (already used by libnftables, > so no extra dependancy) event_driver and nl_driver are forgotten. > NFT_FLAG_NL_ASYNC: Same as previous but a event_driver is at least > mandatory. > And a nl_driver can be provided to support libnl, others... Also commented this with Patrick and I don't think we need this driver infrastructure. We get nothing from supporting two different libraries as drivers. All your efforts should focus on libnftables. > I am thinking of those as well: > > NFT_FLAG_FOLLOW_LOCAL (default): will notice about the locally manipulated > rules events. > NFT_FLAG_FOLLOW_GLOBAL: will keep notice about the whole rule set event. Look at existing netfilter libraries like libnftables. All features should be set via some set/get API, so we won't have hard times to introduce new tweak and features. > Return: 0 on success, negative error value instead. > > void nft_exit(void): > -> exit the library. > > > Basic operations: > ================ > > Signature of callback has to be designed further. Event type, handle, > user data, ... > > int nft_set_notification_callback(nft_notification_callback_f): > -> notify on nftables event (NFT_FLAG_FOLLOW_* applies) > > int nft_execute_statement(nft_statement_callback_f, void *user_data, > char *statement_format, ...): It's very important that you expose the file descriptor that allows you to operate with netlink. We provide elaborated abstractions in the past that were hidding this details, and they were all leaky abstractions in some aspects. The more control your provide to the client of your library on the communication with the kernel, the best. So you can probably have something like _send() and _recv() like functions that return high level objects rather that callbacks that conceal the internal behaviour. You can also define higher level function that do all in once for lazy people, but providing different levels of abstractions is generally good. > -> execute statement. Execution flow depends on NFT_FLAG_NL_* of course. > This could be compared to what is done currently with iptables commands > thrown through a sub-process. But here, no sub-process, notifications, ... > > > Context based: > ============= > > When nft_execute_statement() is nice for one-shot call and forget, > some applications might want to follow precisely what's happening in > a long-run on nftables rule-set either in general or depending on > specific context which can be plural in same application. One might > need to temporarily set a certain number of rules dedicated to a > situation. This part of the API will help on that. Not sure I understand this context based thing. It can help if you provide some usecase for it. > From internal execution point of view, it will be transaction based. > > Callback signature will probably follow the same as previous ones, with > slight difference like adding the context pointer to it. > > struct nft_ctx *nft_ctx_new(nft_context_callback_f) > -> create a new context. > > int nft_ctx_list(struct nft_ctx *ctx, FILE *out) > -> list the context's owned statements > > int nft_ctx_enable(struct nft_ctx *ctx) > -> apply statements previously executed > > int nft_ctx_disable(struct nft_ctx *ctx) > -> delete statements previously executed > > void nft_ctx_free(struct nft_ctx *ctx) > -> free the context (do not enable/disable anything here) > > int nft_ctx_execute_statement(struct nft_ctx *ctx, > char *statement_format, ...) > -> execute the statement related to given context. > > > Listing: > ======= > > int nft_setup_list_driver(int flags, > struct nft_list_driver *driver): > -> set the output format for listing the statements > flags: NFT_LIST_DRIVER_XML (current default format in libnftables) > NFT_LIST_DRIVER_JSON (that format idea comes from Julien Vehent) > NFT_LIST_DRIVER_EXTERNAL -> driver should be provided (need small > modifications in libnftables) How this external driver thing would look like? > int nft_list_all(FILE *out) > > We could have specific things like nft_list_by_handle()... not sure > if it will be useful. > > Besides, we could have lower level helpers, like nft_table_new(char > *table_name) or whatever comes. If relevant, only. > > If you have comments and/or ideas, please go ahead. I am anyway starting to > implement itnowand I will in parallel create real tools to test it according > to specific use cases. > > @Pablo: could you create a repository on netfilter.org's git? libnft? > > or whatever comes as better name if anybody has a nice proposition I think a good way to see the API proposal is to write a batch of use-case example code. So we can all see how the workflow with the library will look like before any real library code is written. Regards. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-19 10:05 ` Pablo Neira Ayuso @ 2013-04-19 11:26 ` Tomasz Bursztyka 2013-04-19 12:11 ` Pablo Neira Ayuso 2013-04-22 20:05 ` Jesper Dangaard Brouer 1 sibling, 1 reply; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-19 11:26 UTC (permalink / raw) To: Pablo Neira Ayuso Cc: Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent Hi Pablo, >> I am thinking of taking original nft tool's lexer/parser for the statements. >> It's nice, it works, no need to reinvent the wheel. The change will be on >> creating libnftables objects instead. > We can just export the abstract syntax tree functions to build the > rules. I am not sure about what you mean. To me, we let the user playing with the same exact statement they will use through nft tool. > > I've also discussing with Patrick that some even higher level API, > something simple for users, just like "match tcp port 22" would be > good to have. Wait, you want to introduce another statement format? How is it going to be simpler than nft tool format? I don't get the point here. >> Flags: >> NFT_FLAG_NL_SYNC (default): using libmnl (already used by libnftables, >> so no extra dependancy) event_driver and nl_driver are forgotten. >> NFT_FLAG_NL_ASYNC: Same as previous but a event_driver is at least >> mandatory. >> And a nl_driver can be provided to support libnl, others... > Also commented this with Patrick and I don't think we need this driver > infrastructure. We get nothing from supporting two different > libraries as drivers. All your efforts should focus on libnftables. It's a requirement for me to be able to use another way to discuss on netlink. Let's say it: something else than libmnl or libnl. And it's not that complicated anyway to handle, as for the event loop actually. I am definitely keeping that. And libnftables has nothing to do with netlink socket layer or event loop. Which is actually very good. In fact libntables will be widely used by the library core. For parsing/building nlh message, maintaining the cache. No other lib but this one on those task. Btw on the cache feature, it will require some tweaks, since the way libnftables keeps track of objects is not efficient currently: - no specific data (void *) can be attached to any objects (easy fix) - and table chain and rule have no better relationship on the object model than the name. For instance if you want to retrieve all the rules of one chain you need to go through the whole rule list and compare the chain name of each rule. For that later one, maybe we can stick on the void* data addition to each object structure, and let its usage as we want externally from libnftables. on libnft we could attach the specific table's chain list to it for instance, same for chain's rule list etc... >> I am thinking of those as well: >> >> NFT_FLAG_FOLLOW_LOCAL (default): will notice about the locally manipulated >> rules events. >> NFT_FLAG_FOLLOW_GLOBAL: will keep notice about the whole rule set event. > Look at existing netfilter libraries like libnftables. All features > should be set via some set/get API, so we won't have hard times to > introduce new tweak and features. That as nothing to do with libnftables set/get stuff. This is for the application notification. In short it tries to answer this: Should it be notified for all nftables event or only about own statements? Some kind of notification filtering. Might be handled another way though. Ideas are welcome. >> Signature of callback has to be designed further. Event type, handle, >> user data, ... >> >> int nft_set_notification_callback(nft_notification_callback_f): >> -> notify on nftables event (NFT_FLAG_FOLLOW_* applies) >> >> int nft_execute_statement(nft_statement_callback_f, void *user_data, >> char *statement_format, ...): > It's very important that you expose the file descriptor that allows > you to operate with netlink. We provide elaborated abstractions in the > past that were hidding this details, and they were all leaky > abstractions in some aspects. > > The more control your provide to the client of your library on the > communication with the kernel, the best. So you can probably have > something like _send() and _recv() like functions that return high > level objects rather that callbacks that conceal the internal > behaviour. You can also define higher level function that do all in > once for lazy people, but providing different levels of abstractions > is generally good. I am fine with that, providing different levels of abstraction. But I am pretty sure people will prefer using higher level one: see how applications uses iptables tool for instance? Raising a subprocess, it is simple (and crap, yes). I bet these people would like to get something almost as simple. My point is to provide something that simple, yet much better (no sub-process, handling notifications... thanks to nftables netlink api) without the requirement for devs to deal with netlink stuff by themselves. But why not exporting so core hooks indeed, can't harm. Let's see. >> Context based: >> ============= >> >> When nft_execute_statement() is nice for one-shot call and forget, >> some applications might want to follow precisely what's happening in >> a long-run on nftables rule-set either in general or depending on >> specific context which can be plural in same application. One might >> need to temporarily set a certain number of rules dedicated to a >> situation. This part of the API will help on that. > Not sure I understand this context based thing. It can help if you > provide some usecase for it. Let's say you have an application listening to users logging in. And when it happens it sets a bunch of dedicated rules for each users. When disconnecting, instead of coding one by one "delete this rule, this one too etc..." you just disable and free the context. On nftables notification (i.e. one rule gets deleted by other app), it could retrieve quickly to which context so to which user it belongs, and it could retry inserting this rule again. Things like that. Just an example. Of course this could be done in the application itself, but it might be nice to provide such abstraction already in the lib. Well, let's put it aside for now. There is already a lot to do first. >> Listing: >> ======= >> >> int nft_setup_list_driver(int flags, >> struct nft_list_driver *driver): >> -> set the output format for listing the statements >> flags: NFT_LIST_DRIVER_XML (current default format in libnftables) >> NFT_LIST_DRIVER_JSON (that format idea comes from Julien Vehent) >> NFT_LIST_DRIVER_EXTERNAL -> driver should be provided (need small >> modifications in libnftables) > How this external driver thing would look like? That has to be determined. Something like { void dump_table(buffer, length, table); //buffer or stream? void dump_chain(..., chain); void dump_rule(..., rule); } It would work on libnftables objects. I haven't thought much about it, but it's required to be able to dump the manipulated rule-set. > I think a good way to see the API proposal is to write a batch of > use-case example code. So we can all see how the workflow with the > library will look like before any real library code is written. I can prepare some. Br, Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-19 11:26 ` Tomasz Bursztyka @ 2013-04-19 12:11 ` Pablo Neira Ayuso 2013-04-22 23:03 ` Eric Leblond 2013-04-23 10:15 ` Tomasz Bursztyka 0 siblings, 2 replies; 20+ messages in thread From: Pablo Neira Ayuso @ 2013-04-19 12:11 UTC (permalink / raw) To: Tomasz Bursztyka Cc: Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent On Fri, Apr 19, 2013 at 02:26:16PM +0300, Tomasz Bursztyka wrote: > Hi Pablo, > > >>I am thinking of taking original nft tool's lexer/parser for the statements. > >>It's nice, it works, no need to reinvent the wheel. The change will be on > >>creating libnftables objects instead. > > >We can just export the abstract syntax tree functions to build the > >rules. > > I am not sure about what you mean. To me, we let the user playing > with the same exact statement they will use through nft tool. After the parsing, nft generates and abstract syntax tree (see list of statements and struct expr for instance). You can use that do build the rule-set. You can easily build rules using that tree structure. > >I've also discussing with Patrick that some even higher level API, > >something simple for users, just like "match tcp port 22" would be > >good to have. > > Wait, you want to introduce another statement format? How is it > going to be simpler than nft tool format? I don't get the point > here. > > >>Flags: > >>NFT_FLAG_NL_SYNC (default): using libmnl (already used by libnftables, > >>so no extra dependancy) event_driver and nl_driver are forgotten. > >>NFT_FLAG_NL_ASYNC: Same as previous but a event_driver is at least > >>mandatory. > >>And a nl_driver can be provided to support libnl, others... > > >Also commented this with Patrick and I don't think we need this driver > >infrastructure. We get nothing from supporting two different > >libraries as drivers. All your efforts should focus on libnftables. > > It's a requirement for me to be able to use another way to discuss > on netlink. Let's say it: something else than libmnl or libnl. > And it's not that complicated anyway to handle, as for the event > loop actually. I am definitely keeping that. I don't get why you may need something different than libmnl or libnl. > And libnftables has nothing to do with netlink socket layer or event > loop. Which is actually very good. > In fact libntables will be widely used by the library core. For > parsing/building nlh message, maintaining the cache. No other lib > but this one on those task. > > Btw on the cache feature, it will require some tweaks, since the way > libnftables keeps track of objects is not efficient currently: > - no specific data (void *) can be attached to any objects (easy fix) > - and table chain and rule have no better relationship on the object > model than the name. > For instance if you want to retrieve all the rules of one chain you > need to go through the whole rule list and compare the chain name of > each rule. That's very easy to fix in the kernel. You only need a small patch for nf_tables_api.c to dump only rules attached to one chain. > For that later one, maybe we can stick on the void* data addition to > each object structure, and let its usage as we want externally from > libnftables. What kind of extra information you need to attach to objects? > on libnft we could attach the specific table's chain list to it for > instance, same for chain's rule list etc... > > >>I am thinking of those as well: > >> > >>NFT_FLAG_FOLLOW_LOCAL (default): will notice about the locally manipulated > >>rules events. > >>NFT_FLAG_FOLLOW_GLOBAL: will keep notice about the whole rule set event. > >Look at existing netfilter libraries like libnftables. All features > >should be set via some set/get API, so we won't have hard times to > >introduce new tweak and features. > > That as nothing to do with libnftables set/get stuff. > > This is for the application notification. In short it tries to > answer this: Should it be notified for all nftables event or only > about own statements? What I'm trying to explain is that all these tweaks and user preferences should be done by set/get API similar to setsockopt/getsockopt. Not as flags to the main nft_init function. > Some kind of notification filtering. Might be handled another way > though. Ideas are welcome. > > >>Signature of callback has to be designed further. Event type, handle, > >>user data, ... > >> > >>int nft_set_notification_callback(nft_notification_callback_f): > >>-> notify on nftables event (NFT_FLAG_FOLLOW_* applies) > >> > >>int nft_execute_statement(nft_statement_callback_f, void *user_data, > >> char *statement_format, ...): > > > >It's very important that you expose the file descriptor that allows > >you to operate with netlink. We provide elaborated abstractions in the > >past that were hidding this details, and they were all leaky > >abstractions in some aspects. > > > >The more control your provide to the client of your library on the > >communication with the kernel, the best. So you can probably have > >something like _send() and _recv() like functions that return high > >level objects rather that callbacks that conceal the internal > >behaviour. You can also define higher level function that do all in > >once for lazy people, but providing different levels of abstractions > >is generally good. > > I am fine with that, providing different levels of abstraction. But > I am pretty sure people will prefer using higher level one: see how > applications uses iptables tool for instance? Don't be so sure about that. People making very simple applications will stick to the simple make-my-life-easy API, yes. But people willing to make more advanced stuff will end up requiring to access details at different levels of the abstraction. > Raising a subprocess, it is simple (and crap, yes). I bet these > people would like to get something almost as simple. My point is to > provide something that simple, yet much better (no sub-process, > handling notifications... thanks to nftables netlink api) without > the requirement for devs to deal with netlink stuff by themselves. > But why not exporting so core hooks indeed, can't harm. Let's see. > > >>Context based: > >>============= > >> > >>When nft_execute_statement() is nice for one-shot call and forget, > >>some applications might want to follow precisely what's happening in > >>a long-run on nftables rule-set either in general or depending on > >>specific context which can be plural in same application. One might > >>need to temporarily set a certain number of rules dedicated to a > >>situation. This part of the API will help on that. > > > >Not sure I understand this context based thing. It can help if you > >provide some usecase for it. > > Let's say you have an application listening to users logging in. And > when it happens it sets a bunch of dedicated rules for each users. > When disconnecting, instead of coding one by one "delete this rule, > this one too etc..." you just disable and free the context. > On nftables notification (i.e. one rule gets deleted by other app), > it could retrieve quickly to which context so to which user it > belongs, and it could retry inserting this rule again. > Things like that. Just an example. I see, thanks. The use-case makes sense to me. > Of course this could be done in > the application itself, but it might be nice to provide such > abstraction already in the lib. > > Well, let's put it aside for now. There is already a lot to do first. > > >>Listing: > >>======= > >> > >>int nft_setup_list_driver(int flags, > >> struct nft_list_driver *driver): > >>-> set the output format for listing the statements > >>flags: NFT_LIST_DRIVER_XML (current default format in libnftables) > >> NFT_LIST_DRIVER_JSON (that format idea comes from Julien Vehent) > >> NFT_LIST_DRIVER_EXTERNAL -> driver should be provided (need small > >> modifications in libnftables) > >How this external driver thing would look like? > > That has to be determined. Something like > { > void dump_table(buffer, length, table); //buffer or stream? > void dump_chain(..., chain); > void dump_rule(..., rule); > } > > It would work on libnftables objects. > I haven't thought much about it, but it's required to be able to > dump the manipulated rule-set. > > >I think a good way to see the API proposal is to write a batch of > >use-case example code. So we can all see how the workflow with the > >library will look like before any real library code is written. > > I can prepare some. Good. Regards. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-19 12:11 ` Pablo Neira Ayuso @ 2013-04-22 23:03 ` Eric Leblond 2013-04-22 23:50 ` Pablo Neira Ayuso 2013-04-23 10:15 ` Tomasz Bursztyka 1 sibling, 1 reply; 20+ messages in thread From: Eric Leblond @ 2013-04-22 23:03 UTC (permalink / raw) To: Pablo Neira Ayuso Cc: Tomasz Bursztyka, Netfilter Development Mailing list, Patrick McHardy, Julien Vehent Hi, On Fri, 2013-04-19 at 14:11 +0200, Pablo Neira Ayuso wrote: > On Fri, Apr 19, 2013 at 02:26:16PM +0300, Tomasz Bursztyka wrote: > > Hi Pablo, > > > > >>I am thinking of taking original nft tool's lexer/parser for the statements. > > >>It's nice, it works, no need to reinvent the wheel. The change will be on > > >>creating libnftables objects instead. > > > > >We can just export the abstract syntax tree functions to build the > > >rules. > > > > I am not sure about what you mean. To me, we let the user playing > > with the same exact statement they will use through nft tool. > > After the parsing, nft generates and abstract syntax tree (see list of > statements and struct expr for instance). You can use that do build > the rule-set. You can easily build rules using that tree structure. > > > >I've also discussing with Patrick that some even higher level API, > > >something simple for users, just like "match tcp port 22" would be > > >good to have. > > > > Wait, you want to introduce another statement format? How is it > > going to be simpler than nft tool format? I don't get the point > > here. > > > > >>Flags: > > >>NFT_FLAG_NL_SYNC (default): using libmnl (already used by libnftables, > > >>so no extra dependancy) event_driver and nl_driver are forgotten. > > >>NFT_FLAG_NL_ASYNC: Same as previous but a event_driver is at least > > >>mandatory. > > >>And a nl_driver can be provided to support libnl, others... > > > > >Also commented this with Patrick and I don't think we need this driver > > >infrastructure. We get nothing from supporting two different > > >libraries as drivers. All your efforts should focus on libnftables. > > > > It's a requirement for me to be able to use another way to discuss > > on netlink. Let's say it: something else than libmnl or libnl. > > And it's not that complicated anyway to handle, as for the event > > loop actually. I am definitely keeping that. > > I don't get why you may need something different than libmnl or libnl. I don't think Tomasz need it but future application developers will really love to avoid to learn how netlink messages are build. The application developers are often doing both the GUI work (with crazy things like web framework, QT or GTK) and they will benefit in using a simple and consistent API where details of kernel-userspace interaction will be transparently handled by the library. The current "let's fork an iptables" approach was heavily used because it was really simple to implement. The main point was that only the command line syntax has to be known. So I think using the nft grammar can be a good idea. At least for the basic insertion/modification/deletion operations. ... > > I am fine with that, providing different levels of abstraction. But > > I am pretty sure people will prefer using higher level one: see how > > applications uses iptables tool for instance? > > Don't be so sure about that. People making very simple applications > will stick to the simple make-my-life-easy API, yes. But people > willing to make more advanced stuff will end up requiring to access > details at different levels of the abstraction. We definitely need different level of abstractions. At least a high level and a low level. The high level abstraction should be almost nftables independent (and netlink independent for sure) and the lower level can get into the details. I think Jesper's mail has described some features that would be really useful in the high level API. Maybe continuing the discussion on these type of features will lead to the list of the thing that would be useful for most people. <idea type=stupid>Regarding the low level API, I did not yet look a libnftables but maybe just keeping it could be ok ?</idea> BR, -- Eric Leblond <eric@regit.org> Blog: https://home.regit.org/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-22 23:03 ` Eric Leblond @ 2013-04-22 23:50 ` Pablo Neira Ayuso 2013-04-23 10:15 ` Tomasz Bursztyka 0 siblings, 1 reply; 20+ messages in thread From: Pablo Neira Ayuso @ 2013-04-22 23:50 UTC (permalink / raw) To: Eric Leblond Cc: Tomasz Bursztyka, Netfilter Development Mailing list, Patrick McHardy, Julien Vehent Hi, On Tue, Apr 23, 2013 at 01:03:11AM +0200, Eric Leblond wrote: [...] > > > It's a requirement for me to be able to use another way to discuss > > > on netlink. Let's say it: something else than libmnl or libnl. > > > And it's not that complicated anyway to handle, as for the event > > > loop actually. I am definitely keeping that. > > > > I don't get why you may need something different than libmnl or libnl. > > I don't think Tomasz need it but future application developers will > really love to avoid to learn how netlink messages are build. I was referring to the driver switch that he wants to introduce to select libnl or libmnl in runtime, it makes no sense to me. [...] > > > I am fine with that, providing different levels of abstraction. But > > > I am pretty sure people will prefer using higher level one: see how > > > applications uses iptables tool for instance? > > > > Don't be so sure about that. People making very simple applications > > will stick to the simple make-my-life-easy API, yes. But people > > willing to make more advanced stuff will end up requiring to access > > details at different levels of the abstraction. > > We definitely need different level of abstractions. Nobody said here that we don't need high level abstractions. My point is that the library should provide different level of abstractions, so you can switch to the level you want. Think of it as an elevator that takes you to the 1st floor or 9th floor. Depending where you get, you see more or less details. If designed carefully, that should be possible. > At least a high level and a low level. The high level abstraction > should be almost nftables independent (and netlink independent for > sure) and the lower level can get into the details. > > I think Jesper's mail has described some features that would be really > useful in the high level API. Maybe continuing the discussion on these > type of features will lead to the list of the thing that would be useful > for most people. I think you misundertood my use-case request. I'm not asking for more features or ideas. There are tons of cool things we can do with nftables, all those mentioned should be possible with some manpower working on those. Instead, I'm asking for simple source examples, eg. one to add a rule, one to dump the entire rule-set, one to listen to events, etc. how the objects would look like, what workflow is imposed to the programmer, and so on. All those using the hypothetical new high level API. Those should example files of the proposed API should help to see how that looks like, even if there is not real library code available for those yet. > <idea type=stupid>Regarding the low level API, I did not yet look a > libnftables but maybe just keeping it could be ok ?</idea> Of course it would be good idea to keep it. It does not make any sense to me to force people to use one single high level API. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-22 23:50 ` Pablo Neira Ayuso @ 2013-04-23 10:15 ` Tomasz Bursztyka 2013-04-23 11:31 ` Pablo Neira Ayuso 0 siblings, 1 reply; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-23 10:15 UTC (permalink / raw) To: Pablo Neira Ayuso Cc: Eric Leblond, Netfilter Development Mailing list, Patrick McHardy, Julien Vehent Hi Eric and Pablo, >>> I don't get why you may need something different than libmnl or libnl. >> I don't think Tomasz need it but future application developers will >> really love to avoid to learn how netlink messages are build. I actually need this flexibility. But indeed, it's thought to hide netlink IOs anyway. > I was referring to the driver switch that he wants to introduce to > select libnl or libmnl in runtime, it makes no sense to me. It's easy to support it. Moreover that it won't affect all devs: most will just initiate the library to use libnl or libmnl, and won't have to provide their own driver implementation. They will just call nft_init() with NULL as the nl driver and that's it. If it does not make sense to you to bring this little flexibility then ok, let's use libnl. But then no need to use limnl/libnftables: let's use instead libnl-nftables which already does everything on this level, afaik. My point of using libmnl/libnftables was especially to bring this possibility of using something else than libmnl/libnl to discuss with netlink: since libnftables it dedicated on parsing/building nftables nlh (it's perfect for that), and since libnftables has a dependency over libmnl, let's use libmnl as the default one for netlink conversation. We could solve it another way, like exposing core functions, the fds and so on, like libdbus does but I don't like it much. Instead, providing a structure which tells what subset of functions and their signatures is required is nicer. As I said, I tried a quick PoC on that. And you can see that feature as being part of a lower level access in the API. >> We definitely need different level of abstractions. > Nobody said here that we don't need high level abstractions. > > My point is that the library should provide different level of > abstractions, so you can switch to the level you want. Think of it as > an elevator that takes you to the 1st floor or 9th floor. Depending > where you get, you see more or less details. If designed carefully, > that should be possible. I am not against such different levels, but indeed it requires to be careful: if we propose too much things without good reasons, many devs will just miss-use the API for sure. I guess we should provide this different levels during the lib implementation. It is easier to solve this from top level to bottom, incrementally. Br, Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-23 10:15 ` Tomasz Bursztyka @ 2013-04-23 11:31 ` Pablo Neira Ayuso 2013-04-23 11:55 ` Tomasz Bursztyka 0 siblings, 1 reply; 20+ messages in thread From: Pablo Neira Ayuso @ 2013-04-23 11:31 UTC (permalink / raw) To: Tomasz Bursztyka Cc: Eric Leblond, Netfilter Development Mailing list, Patrick McHardy, Julien Vehent On Tue, Apr 23, 2013 at 01:15:10PM +0300, Tomasz Bursztyka wrote: > Hi Eric and Pablo, > >>>I don't get why you may need something different than libmnl or libnl. > >> > >>I don't think Tomasz need it but future application developers will > >>really love to avoid to learn how netlink messages are build. > > I actually need this flexibility. But indeed, it's thought to hide > netlink IOs anyway. [...] > >I was referring to the driver switch that he wants to introduce to > >select libnl or libmnl in runtime, it makes no sense to me. > > It's easy to support it. [...] Again, you say you need this driver thing, but you will not escape without explaining me why you need this. Regards. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-23 11:31 ` Pablo Neira Ayuso @ 2013-04-23 11:55 ` Tomasz Bursztyka 0 siblings, 0 replies; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-23 11:55 UTC (permalink / raw) To: Pablo Neira Ayuso Cc: Eric Leblond, Netfilter Development Mailing list, Patrick McHardy, Julien Vehent Hi Pablo, >> It's easy to support it. > [...] > > Again, you say you need this driver thing, but you will not escape > without explaining me why you need this. We just don't use libnl/libmnl in connman. As well as we currently support glib as the event loop but I know other app devs don't especially, they might use some others. Which is why I propose these 2 features. Br, Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-19 12:11 ` Pablo Neira Ayuso 2013-04-22 23:03 ` Eric Leblond @ 2013-04-23 10:15 ` Tomasz Bursztyka 1 sibling, 0 replies; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-23 10:15 UTC (permalink / raw) To: Pablo Neira Ayuso Cc: Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent Hi Pablo, >> >> I am not sure about what you mean. To me, we let the user playing >> with the same exact statement they will use through nft tool. > After the parsing, nft generates and abstract syntax tree (see list of > statements and struct expr for instance). You can use that do build > the rule-set. You can easily build rules using that tree structure. Sound relevant. Actually it might fix one of Jesper's use cases to export such functions. >> Btw on the cache feature, it will require some tweaks, since the way >> libnftables keeps track of objects is not efficient currently: >> - no specific data (void *) can be attached to any objects (easy fix) >> - and table chain and rule have no better relationship on the object >> model than the name. >> For instance if you want to retrieve all the rules of one chain you >> need to go through the whole rule list and compare the chain name of >> each rule. > That's very easy to fix in the kernel. You only need a small patch for > nf_tables_api.c to dump only rules attached to one chain. You got me wrong here. When handling a cache (which gets updated via nftables notifications, after a full dump), the point is to query it easily without the need of kernel calls. Like nft_is_chain_present(my_chaine_name), things like that... (just a quick example) In fact, we should lower as much as possible the kernel calls. Currently, the way libnftables stores the objects is under performing. As I said if you want to quickly retrieve all rules of 1 chain, you have no other possibility right now go through the rule list and compare the chain name. Instead if it would be possible to add our own pointer in a chain (user data then) we could add the rule list pointer in the chain. So all chain objects would own a pointer on their rule list. It's a q&d idea, we can solve it another way. And for tables and chain (mostly chains actually), we could store them in a hash table: key as the name, object pointer as a value. Some implementation details like that. Let's figure it out when implementing the lib. >>>> I am thinking of those as well: >>>> >>>> NFT_FLAG_FOLLOW_LOCAL (default): will notice about the locally manipulated >>>> rules events. >>>> NFT_FLAG_FOLLOW_GLOBAL: will keep notice about the whole rule set event. >>> Look at existing netfilter libraries like libnftables. All features >>> should be set via some set/get API, so we won't have hard times to >>> introduce new tweak and features. >> That as nothing to do with libnftables set/get stuff. >> >> This is for the application notification. In short it tries to >> answer this: Should it be notified for all nftables event or only >> about own statements? > What I'm trying to explain is that all these tweaks and user > preferences should be done by set/get API similar to > setsockopt/getsockopt. Not as flags to the main nft_init function. Still: libnftables does not do anything with netlink send/recv, notifications, etc... this is not libnftables related. For me libnftables is only a nlh parser/builder, and it does it very well. Unless you have plans to broaden its features scope? Anyway let's put aside such flags, let's fix the first thing first: executing nft statement and handle notifications. It will help figuring out advanced features then. Br, Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-19 10:05 ` Pablo Neira Ayuso 2013-04-19 11:26 ` Tomasz Bursztyka @ 2013-04-22 20:05 ` Jesper Dangaard Brouer 2013-04-22 22:26 ` Eric Leblond ` (2 more replies) 1 sibling, 3 replies; 20+ messages in thread From: Jesper Dangaard Brouer @ 2013-04-22 20:05 UTC (permalink / raw) To: Pablo Neira Ayuso Cc: Tomasz Bursztyka, Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent, Fabio Di Nitto, Jiri Benc, Daniel Borkmann, Thomas Graf First of all, thanks Tomasz for proposing to write a high level API for nftables. Note to cc'ed people not on the netfilter-devel list can follow the thread here: http://thread.gmane.org/gmane.comp.security.firewalls.netfilter.devel/46734 On Fri, 2013-04-19 at 12:05 +0200, Pablo Neira Ayuso wrote: > I think a good way to see the API proposal is to write a batch of > use-case example code. So we can all see how the workflow with the > library will look like before any real library code is written. Use-case 1 ---------- At ComX Networks, I needed to build a "SubnetSkeleton" tree structure with iptables (https://github.com/netoptimizer/IPTables-SubnetSkeleton/blob/master/lib/IPTables/SubnetSkeleton.pm#L440) For this I needed some API calls, to query if some rules and chains already existed. There was an API for testing if a chain existed, I used when building the tree. And the assumed that the jump rule in/to the chain was correct, as no API existed for asking if a rule existed, To avoid inserting a rule twice, I solved this by the hack of simply first delete the rule, and the insert the rule. I would really have liked a test if rule exist API instead. Use-case 2 ----------- Think this was Fabio's use-case during the netfilter workshop. An interface to dry run a packet through configured netfilter policy. This would allow user space to figure out if a specific daemon or use-case can function in the configured environment. The feature is primarily intended for debugging and troubleshooting purposes but can be extended later on, enabling daemons or daemon management tools to verify if the daemon is permitted to run in the configured specific environment. I guess, we also would need some kernel changes for supporting this? Use-case 3 ---------- Related to use-case 2. Have iptables issue a warning if a new rule would prohibit a well known service from functioning. We could use the notification system in nftables to get notified about some rule changed occurred. Then we could use the API from use-case 2, to query if our service is still allowed to work. -- Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-22 20:05 ` Jesper Dangaard Brouer @ 2013-04-22 22:26 ` Eric Leblond 2013-04-23 7:27 ` Fabio M. Di Nitto 2013-04-23 10:15 ` Tomasz Bursztyka 2 siblings, 0 replies; 20+ messages in thread From: Eric Leblond @ 2013-04-22 22:26 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Pablo Neira Ayuso, Tomasz Bursztyka, Netfilter Development Mailing list, Patrick McHardy, Julien Vehent, Fabio Di Nitto, Jiri Benc, Daniel Borkmann, Thomas Graf Hi, I agree on all points of this mail. Some comments below. On Mon, 2013-04-22 at 22:05 +0200, Jesper Dangaard Brouer wrote: > First of all, thanks Tomasz for proposing to write a high level API for > nftables. +1 > Note to cc'ed people not on the netfilter-devel list can follow the > thread here: > http://thread.gmane.org/gmane.comp.security.firewalls.netfilter.devel/46734 ... > Use-case 2 > ----------- > Think this was Fabio's use-case during the netfilter workshop. > > An interface to dry run a packet through configured netfilter policy. > > This would allow user space to figure out if a specific daemon or > use-case can function in the configured environment. > > The feature is primarily intended for debugging and troubleshooting > purposes but can be extended later on, enabling daemons or daemon > management tools to verify if the daemon is permitted to run in the > configured specific environment. > > I guess, we also would need some kernel changes for supporting this? I think this point is really interesting. Being able to know for a given packet (IP tuple + ifaces)if it will get dropped or accepted or NAted (and with which transformation) could be really interesting. A more advanced related feature could be to have a TRACE like result . By the way, I've encountered today a TRACE limitation related to this point when debugging a firewall. I was investigating a NAT issue relative to a REJECT rule and TRACE was not tracing the sent ICMP message. And as for result, the NAT transformation made on the error message was not visible. This kind of information would be really useful if the test system is implemented. BR, -- Eric Leblond <eric@regit.org> Blog: https://home.regit.org/ ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-22 20:05 ` Jesper Dangaard Brouer 2013-04-22 22:26 ` Eric Leblond @ 2013-04-23 7:27 ` Fabio M. Di Nitto 2013-04-23 10:15 ` Tomasz Bursztyka 2 siblings, 0 replies; 20+ messages in thread From: Fabio M. Di Nitto @ 2013-04-23 7:27 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Pablo Neira Ayuso, Tomasz Bursztyka, Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent, Jiri Benc, Daniel Borkmann, Thomas Graf Hi Jesper, thanks for summarizing it and keep me in the loop. On 4/22/2013 10:05 PM, Jesper Dangaard Brouer wrote: Use-case 4 ---------- I discussed this with Pablo after the conference (well I rather cornered him and forced him to listen to me ;)). Common per socket ACL re-using in kernel nftable. Problem: virtually all userland daemons implement some kind of access list (source ip, interface, etc.). This is a lot of duplicated code across many implementations that might be not as efficient as they could be and not as flexible. Discussion lead to have some kind of per socket nftable that could (for example): 1) drop packets directly (saves a few memcpy down to userland and userland processing). 2) mark the packet as "this one matched rule X or Y" and send the data down to userland in the ancillary data, to allow userland to warn/take other actions. and from a troubleshooting/debugging perspective, by having all those rules in the place (nftable/kernel), would allow a user to: nftable --fulldump (whatever option) and see effectively all the rules that apply to the host and the applications in one go, without having to interact with multiple parts of the system (first nftable, then per application config files, etc). An approach of this kind also integrates very well with use-case 2 and 3 because effectively, a dry run would be able to navigate all the active rules at once. Thanks again for taking time to listen to my rumbling guys! Fabio ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-22 20:05 ` Jesper Dangaard Brouer 2013-04-22 22:26 ` Eric Leblond 2013-04-23 7:27 ` Fabio M. Di Nitto @ 2013-04-23 10:15 ` Tomasz Bursztyka 2013-04-23 18:49 ` Jesper Dangaard Brouer 2 siblings, 1 reply; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-23 10:15 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Pablo Neira Ayuso, Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent, Fabio Di Nitto, Jiri Benc, Daniel Borkmann, Thomas Graf Hi Jesper, > First of all, thanks Tomasz for proposing to write a high level API for > nftables. We all need this and I know I will not be alone implementing it. > Use-case 1 > ---------- > At ComX Networks, I needed to build a "SubnetSkeleton" tree structure > with iptables > (https://github.com/netoptimizer/IPTables-SubnetSkeleton/blob/master/lib/IPTables/SubnetSkeleton.pm#L440) > > For this I needed some API calls, to query if some rules and chains > already existed. There was an API for testing if a chain existed, I > used when building the tree. And the assumed that the jump rule in/to > the chain was correct, as no API existed for asking if a rule existed, > > To avoid inserting a rule twice, I solved this by the hack of simply > first delete the rule, and the insert the rule. I would really have > liked a test if rule exist API instead. I stumble into the same use case with iptables and indeed it got solved the same way. But since we are dealing with much better kernel stack, we could solve this differently: - either via proposing low level functions requesting the cache as you propose (and we probably should) - and/or when using nft_execute_statement() it would check if it already exist - or not, if it's a delete statement - by itself, hiding the details to the dev and raising a success relevantly. > Use-case 2 > ----------- > Think this was Fabio's use-case during the netfilter workshop. > > An interface to dry run a packet through configured netfilter policy. > > This would allow user space to figure out if a specific daemon or > use-case can function in the configured environment. > > The feature is primarily intended for debugging and troubleshooting > purposes but can be extended later on, enabling daemons or daemon > management tools to verify if the daemon is permitted to run in the > configured specific environment. > > I guess, we also would need some kernel changes for supporting this? Indeed, it needs to be thought and implemented there first. Br, Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-23 10:15 ` Tomasz Bursztyka @ 2013-04-23 18:49 ` Jesper Dangaard Brouer 2013-04-24 6:06 ` Tomasz Bursztyka 0 siblings, 1 reply; 20+ messages in thread From: Jesper Dangaard Brouer @ 2013-04-23 18:49 UTC (permalink / raw) To: Tomasz Bursztyka Cc: Pablo Neira Ayuso, Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent, Fabio Di Nitto, Jiri Benc, Daniel Borkmann, Thomas Graf, Thomas Woerner On Tue, 2013-04-23 at 13:15 +0300, Tomasz Bursztyka wrote: > Hi Jesper, > > > First of all, thanks Tomasz for proposing to write a high level API for > > nftables. > > We all need this and I know I will not be alone implementing it. > > > Use-case 1 > > ---------- > > At ComX Networks, I needed to build a "SubnetSkeleton" tree structure > > with iptables > > (https://github.com/netoptimizer/IPTables-SubnetSkeleton/blob/master/lib/IPTables/SubnetSkeleton.pm#L440) > > > > For this I needed some API calls, to query if some rules and chains > > already existed. There was an API for testing if a chain existed, I > > used when building the tree. And the assumed that the jump rule in/to > > the chain was correct, as no API existed for asking if a rule existed, > > > > To avoid inserting a rule twice, I solved this by the hack of simply > > first delete the rule, and the insert the rule. I would really have > > liked a test if rule exist API instead. > > I stumble into the same use case with iptables and indeed it got solved > the same way. > > But since we are dealing with much better kernel stack, we could solve > this differently: Yes, our current iptables devel API 'libiptc' is broken, and not even officially supported/exported. So, lets create something better this time around. > - either via proposing low level functions requesting the cache as you > propose (and we probably should) I'm a little worried about your idea of implementing a cache. It reminds me of the current libliptc (IPtables Cache) library, which caused a lot of annoying issues in the past. I know, you said we can use the notification system to keep the cache up-to-date, but just have a bad feeling about introducing a cache layer... > - and/or when using nft_execute_statement() it would check if it already > exist - or not, if it's a delete statement - > by itself, hiding the details to the dev and raising a success relevantly. So, your are saying that a nft_execute_statement() that creates a rule sound return a "false" indication if the rule already exists? I would really prefer a real "test/query" operation, but I guess we should extend the kernel API to support this, so we don't need a cache infrastructure for this. > > Use-case 2 > > ----------- > > Think this was Fabio's use-case during the netfilter workshop. > > > > An interface to dry run a packet through configured netfilter policy. > > > > This would allow user space to figure out if a specific daemon or > > use-case can function in the configured environment. > > > > The feature is primarily intended for debugging and troubleshooting > > purposes but can be extended later on, enabling daemons or daemon > > management tools to verify if the daemon is permitted to run in the > > configured specific environment. > > > > I guess, we also would need some kernel changes for supporting this? > > Indeed, it needs to be thought and implemented there first. Yes, some kernel work is required here. -- Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-23 18:49 ` Jesper Dangaard Brouer @ 2013-04-24 6:06 ` Tomasz Bursztyka 2013-04-24 11:23 ` Jesper Dangaard Brouer 0 siblings, 1 reply; 20+ messages in thread From: Tomasz Bursztyka @ 2013-04-24 6:06 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Pablo Neira Ayuso, Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent, Fabio Di Nitto, Jiri Benc, Daniel Borkmann, Thomas Graf, Thomas Woerner Hi Jesper, >> >- either via proposing low level functions requesting the cache as you >> >propose (and we probably should) > I'm a little worried about your idea of implementing a cache. > It reminds me of the current libliptc (IPtables Cache) library, which > caused a lot of annoying issues in the past. > I know, you said we can use the notification system to keep the cache > up-to-date, but just have a bad feeling about introducing a cache > layer... It should be indeed much easier and reliable to maintain a cache with nftables api. But you are right: not all usage might require this. That's why I thought about NFT_FLAG_FOLLOW_* flags One could limit the "caching" towards only application rules/chain/tables (if any), and other one would keep track of everything. >> >- and/or when using nft_execute_statement() it would check if it already >> >exist - or not, if it's a delete statement - >> >by itself, hiding the details to the dev and raising a success relevantly. > So, your are saying that a nft_execute_statement() that creates a rule > sound return a "false" indication if the rule already exists? > I would really prefer a real "test/query" operation, but I guess we > should extend the kernel API to support this, so we don't need a cache > infrastructure for this. I don't know if such existence test should be part of kernel API rather than in user land since when manipulating you need anyway a dump of the rule set. But about nft_execute_statement(), it's not really a false indication: you want to inject a rule which already exist, so it's fine. Or we could handle it so it return an error like -EALREADY if it's preferable. Br, Tomasz ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-24 6:06 ` Tomasz Bursztyka @ 2013-04-24 11:23 ` Jesper Dangaard Brouer 2013-04-24 15:35 ` Stephen Hemminger 0 siblings, 1 reply; 20+ messages in thread From: Jesper Dangaard Brouer @ 2013-04-24 11:23 UTC (permalink / raw) To: Tomasz Bursztyka Cc: Pablo Neira Ayuso, Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent, Fabio Di Nitto, Jiri Benc, Daniel Borkmann, Thomas Graf, Thomas Woerner On Wed, 2013-04-24 at 09:06 +0300, Tomasz Bursztyka wrote: > Hi Jesper, > > >> >- either via proposing low level functions requesting the cache as you > >> >propose (and we probably should) > > I'm a little worried about your idea of implementing a cache. > > It reminds me of the current libliptc (IPtables Cache) library, which > > caused a lot of annoying issues in the past. > > I know, you said we can use the notification system to keep the cache > > up-to-date, but just have a bad feeling about introducing a cache > > layer... > > It should be indeed much easier and reliable to maintain a cache with > nftables api. > But you are right: not all usage might require this. That's why I > thought about NFT_FLAG_FOLLOW_* flags > One could limit the "caching" towards only application > rules/chain/tables (if any), and other one would keep track of everything. > > >> >- and/or when using nft_execute_statement() it would check if it already > >> >exist - or not, if it's a delete statement - > >> >by itself, hiding the details to the dev and raising a success relevantly. > > So, your are saying that a nft_execute_statement() that creates a rule > > sound return a "false" indication if the rule already exists? > > I would really prefer a real "test/query" operation, but I guess we > > should extend the kernel API to support this, so we don't need a cache > > infrastructure for this. > > I don't know if such existence test should be part of kernel API rather > than in user land since when manipulating > you need anyway a dump of the rule set. > > But about nft_execute_statement(), it's not really a false indication: > you want to inject a rule which already exist, so it's fine. > Or we could handle it so it return an error like -EALREADY if it's > preferable. Hmm, yes, I see the problem. It is actually allowed/valid to create many of the exact same rule... We definitely need a "test/query/exists" operation. And it should be fairly simple to implement, as its very similar to a "delete" rule operation, which simply don't actually delete the rule, but just returns if it was "possible" to delete such a rule. (but again a kernel side changes) -- Best regards, Jesper Dangaard Brouer MSc.CS, Sr. Network Kernel Developer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Nftables RFC] High level library proposal 2013-04-24 11:23 ` Jesper Dangaard Brouer @ 2013-04-24 15:35 ` Stephen Hemminger 0 siblings, 0 replies; 20+ messages in thread From: Stephen Hemminger @ 2013-04-24 15:35 UTC (permalink / raw) To: Jesper Dangaard Brouer Cc: Tomasz Bursztyka, Pablo Neira Ayuso, Netfilter Development Mailing list, Patrick McHardy, Eric Leblond, Julien Vehent, Fabio Di Nitto, Jiri Benc, Daniel Borkmann, Thomas Graf, Thomas Woerner On Wed, 24 Apr 2013 13:23:27 +0200 Jesper Dangaard Brouer <brouer@redhat.com> wrote: > On Wed, 2013-04-24 at 09:06 +0300, Tomasz Bursztyka wrote: > > Hi Jesper, > > > > >> >- either via proposing low level functions requesting the cache as you > > >> >propose (and we probably should) > > > I'm a little worried about your idea of implementing a cache. > > > It reminds me of the current libliptc (IPtables Cache) library, which > > > caused a lot of annoying issues in the past. > > > I know, you said we can use the notification system to keep the cache > > > up-to-date, but just have a bad feeling about introducing a cache > > > layer... > > > > It should be indeed much easier and reliable to maintain a cache with > > nftables api. > > But you are right: not all usage might require this. That's why I > > thought about NFT_FLAG_FOLLOW_* flags > > One could limit the "caching" towards only application > > rules/chain/tables (if any), and other one would keep track of everything. > > > > >> >- and/or when using nft_execute_statement() it would check if it already > > >> >exist - or not, if it's a delete statement - > > >> >by itself, hiding the details to the dev and raising a success relevantly. > > > So, your are saying that a nft_execute_statement() that creates a rule > > > sound return a "false" indication if the rule already exists? > > > I would really prefer a real "test/query" operation, but I guess we > > > should extend the kernel API to support this, so we don't need a cache > > > infrastructure for this. > > > > I don't know if such existence test should be part of kernel API rather > > than in user land since when manipulating > > you need anyway a dump of the rule set. > > > > But about nft_execute_statement(), it's not really a false indication: > > you want to inject a rule which already exist, so it's fine. > > Or we could handle it so it return an error like -EALREADY if it's > > preferable. > > Hmm, yes, I see the problem. It is actually allowed/valid to create > many of the exact same rule... > > We definitely need a "test/query/exists" operation. And it should be > fairly simple to implement, as its very similar to a "delete" rule > operation, which simply don't actually delete the rule, but just returns > if it was "possible" to delete such a rule. > (but again a kernel side changes) > Also caching fails badly for the case of a monitoring application. An application that is doing lots of queries ends up getting invalidated by underlying changes. ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2013-04-24 15:35 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-04-17 13:41 [Nftables RFC] High level library proposal Tomasz Bursztyka 2013-04-17 13:52 ` Victor Julien 2013-04-19 6:50 ` Tomasz Bursztyka 2013-04-19 10:05 ` Pablo Neira Ayuso 2013-04-19 11:26 ` Tomasz Bursztyka 2013-04-19 12:11 ` Pablo Neira Ayuso 2013-04-22 23:03 ` Eric Leblond 2013-04-22 23:50 ` Pablo Neira Ayuso 2013-04-23 10:15 ` Tomasz Bursztyka 2013-04-23 11:31 ` Pablo Neira Ayuso 2013-04-23 11:55 ` Tomasz Bursztyka 2013-04-23 10:15 ` Tomasz Bursztyka 2013-04-22 20:05 ` Jesper Dangaard Brouer 2013-04-22 22:26 ` Eric Leblond 2013-04-23 7:27 ` Fabio M. Di Nitto 2013-04-23 10:15 ` Tomasz Bursztyka 2013-04-23 18:49 ` Jesper Dangaard Brouer 2013-04-24 6:06 ` Tomasz Bursztyka 2013-04-24 11:23 ` Jesper Dangaard Brouer 2013-04-24 15:35 ` Stephen Hemminger
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.