Netfilter-Devel Archive on
 help / color / Atom feed
From: "Serguei Bezverkhi (sbezverk)" <>
To: Arturo Borrero Gonzalez <>,
	Phil Sutter <>
Cc: "" <>
Subject: Re: Numen with reference to vmap
Date: Wed, 4 Dec 2019 21:05:31 +0000
Message-ID: <> (raw)
In-Reply-To: <>

Here are code generated  nftables rules for  nat portion of k8s proxy.  Probably it does not cover all cases, but on a normal k8s cluster it would be sufficient. Appreciate reviews and suggestions for optimization. Thank you very much.

table ip ipv4table {
	chain nat-preroutin {
		type nat hook prerouting priority filter; policy accept;
		jump k8s-nat-services

	chain nat-output {
		type nat hook output priority filter; policy accept;
		jump k8s-nat-services

	chain nat-postrouting {
		type nat hook postrouting priority filter; policy accept;
		jump k8s-nat-postrouting

	chain k8s-nat-mark-drop {
		meta mark set 0x00008000

	chain k8s-nat-services {
		ip saddr != ip daddr tcp dport 80 meta mark set 0x00004000
		ip daddr tcp dport 80 jump KUBE-SVC-57XVOCFNTLTR3Q27
		ip saddr != ip daddr tcp dport 15443 meta mark set 0x00004000
		ip daddr tcp dport 15443 jump KUBE-SVC-S4S242M2WNFIAT6Y
		ip daddr tcp dport 8989 jump KUBE-SVC-MUPXPVK4XAZHSWAR
		ip daddr tcp dport 8989 meta mark set 0x00004000
		fib saddr type != local ip daddr tcp dport 8989 iifname != "bridge*" jump KUBE-SVC-MUPXPVK4XAZHSWAR
		fib daddr type local ip daddr tcp dport 8989 jump KUBE-SVC-MUPXPVK4XAZHSWAR

	chain k8s-nat-nodeports {
		tcp dport 30725 meta mark set 0x00004000 jump KUBE-SVC-S4S242M2WNFIAT6Y

	chain k8s-nat-postrouting {
		meta mark 0x00004000 masquerade random,persistent

	chain KUBE-SVC-S4S242M2WNFIAT6Y {

		numgen random mod 2 vmap { 0 : jump KUBE-SEP-FS3FUULGZPVD4VYB, 1 : jump KUBE-SEP-MMFZROQSLQ3DKOQA }


		ip saddr meta mark set 0x00004000
		dnat to fully-random

		ip saddr meta mark set 0x00004000
		dnat to fully-random

		ip saddr meta mark set 0x00004000
		dnat to fully-random

	chain KUBE-SEP-LO6TEVOI6GV524F3 {
		ip saddr meta mark set 0x00004000
		dnat to fully-random

On 2019-12-04, 12:49 PM, "Serguei Bezverkhi (sbezverk)" <> wrote:

    Hello @Phil,
    Just to confirm,
    If I do,
    Numgen random mod 3 vmap { 0  :  jump endpoint1, 1  :  jump endpoint2,  2  :  jump endpoint3 }
    Then if 4th endpoint appears I replace the previous rule with:
    Numgen random mod 4 vmap { 0  :  jump endpoint1, 1  :  jump endpoint2, 2  :  jump endpoint3,  3  :  jump endpoint4 }
    It should do the trick of loadbalancing, right?
    I am no planning to use  " dnat numgen randmo { 0-49 : <ip>:<port> }."
    Each end point will have it is own chain and it will to dnat to ip and specific to endpoint target port. The load balancing will be done in service chain between multiple endpoint chains.
    See example above. Does it make sense?
    Thank you
    On 2019-12-04, 12:31 PM, "Arturo Borrero Gonzalez" <> wrote:
        On 12/4/19 4:56 PM, Phil Sutter wrote:
        > OK, static load-balancing between two services - no big deal. :)
        > What happens if config changes? I.e., if one of the endpoints goes down
        > or a third one is added? (That's the thing we're discussing right now,
        > aren't we?)
        if the non-anon map for random numgen was allowed, then only elements would need
        to be adjusted:
        dnat numgen random mod 100 map { 0-49 :, 50-99 : }
        You could always use mod 100 (or 10000 if you want) and just play with the map
        probabilities by updating map elements. This is a valid use case I think.
        The mod number can just be the max number of allowed endpoints per service in
        I'm not sure if the typeof() thingy will work in this case, since the integer
        length would depend on the mod value used.
        What about introducing something like an explicit u128 integer datatype. Perhaps
        it's useful for other use cases too...
        kubernetes implements a complex chain of mechanisms to deal with traffic. What
        happens if endpoints for a given svc have different ports? I don't know if
        that's supported or not, but then this approach wouldn't work either: you can't
        use dnat numgen randmo { 0-49 : <ip>:<port> }.
        Also, we have the masquerade/drop thing going on too, which needs to be deal
        with and that currently is done by yet another chain jump + packet mark.
        I'm not sure in which state of the development you are, but this is my
        suggestion: Try to don't over-optimize in the first iteration. Just get a
        working nft ruleset with the few optimization that make sense and are easy to
        use (and understand). For iteration #2 we can do better optimizations, including
        patching missing features we may have in nftables.
        I really want a ruleset with very little rules, but we are still comparing with
        the iptables ruleset. I suggest we leave the hard optimization for a later point
        when we are comparing nft vs nft rulesets.

  reply index

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-04  0:54 Serguei Bezverkhi (sbezverk)
2019-12-04 10:18 ` Phil Sutter
2019-12-04 13:47   ` Serguei Bezverkhi (sbezverk)
2019-12-04 15:17     ` Phil Sutter
2019-12-04 15:42       ` Serguei Bezverkhi (sbezverk)
2019-12-04 15:56         ` Phil Sutter
2019-12-04 16:13           ` Serguei Bezverkhi (sbezverk)
2019-12-04 17:00             ` Phil Sutter
2019-12-04 17:31           ` Arturo Borrero Gonzalez
2019-12-04 17:49             ` Serguei Bezverkhi (sbezverk)
2019-12-04 21:05               ` Serguei Bezverkhi (sbezverk) [this message]
2019-12-04 22:32             ` Phil Sutter

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Netfilter-Devel Archive on

Archives are clonable:
	git clone --mirror netfilter-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 netfilter-devel netfilter-devel/ \
	public-inbox-index netfilter-devel

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone