Re: Numen with reference to vmap

["Serguei Bezverkhi (sbezverk)" <sbezverk@cisco.com>]

Hello Phil,

Thank you for your reply. It is very unfortunate indeed. Here is the scenario where I thought to use a non-anonymous vmap.

Each k8s service can have 0, 1 or more associated endpoints, backends (pods providing this service). 0 endpoint already taken care of in filter prerouting hook.  When there are 1 or more, proxy needs to load balance incoming connections between endpoints.I thought to create vmap per service with 1 rule per service . When an endpoint gets updated (add/deleted) which could happen anytime then the only vmap get corresponding update and my hope was that automagically load balancing will be adjusted to use updated endpoints list.

With what you explained, I am not sure if dynamic load balancing is possible at all. If numgen work only with static anonymous vmap and fixed modulus , the only way to address this dynamic nature of endpoints is to recreate service rule everytime when number of endpoints changes (recalculate modulus and entries in vmap). I suspect it is way less efficient.
What will happen to dataplane and packets in transit when the rule will be deleted and then recreated? I suspect it might result in dropped packets, could you please comment on the possible impact?

If you could suggest a better approach for the described scenario, appreciate if you share it.

Thank you
Serguei

On 2019-12-04, 5:18 AM, "n0-1@orbyte.nwl.cc on behalf of Phil Sutter" <n0-1@orbyte.nwl.cc on behalf of phil@nwl.cc> wrote:

    Hi Serguei,
    
    On Wed, Dec 04, 2019 at 12:54:05AM +0000, Serguei Bezverkhi (sbezverk) wrote:
    > Nftables wiki gives this example for numgen:
    > 
    > nft add rule nat prerouting numgen random mod 2 vmap { 0 : jump mychain1, 1 : jump mychain2 }
    > 
    > I would like to use it but with map reference, like this:
    > 
    > nft add rule nat prerouting numgen random mod 2 vmap @service1-endpoints
    > 
    > Could you please confirm if it is supported? If it is what would be the type of the key in such map? I thought it would be integer, but command fails.
    > 
    > sudo nft --debug all add map ipv4table k8s-57XVOCFNTLTR3Q27-endpoints   { type  integer : verdict \; }
    > Error: unqualified key type integer specified in map definition
    > add map ipv4table k8s-57XVOCFNTLTR3Q27-endpoints { type integer : verdict ; }
    >                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
    
    Yes, this is sadly not possible right now. numgen type is 32bit integer,
    but we don't have a type definition matching that. Type 'integer' is
    unqualified regarding size, therefore unsuitable for use in map/set
    definitions.
    
    This all works when using anonymous set/map because key type is
    deduced from map LHS.
    
    We plan to support a 'typeof' keyword at some point to allow for the
    same deduction from within named map/set declarations, but it needs
    further work as the type info is lost on return path (when listing) so
    it would create a ruleset that can't be fed back.
    
    > The ultimate  goal is to update dynamically just the  map  with available endpoints and loadbalance between them without  touching the rule.
    
    I don't quite understand why you need to dynamically change the
    load-balancing rule: numgen modulus is fixed anyway, so the number of
    elements in vmap are fixed. Maybe just jump to chains and dynamically
    update those instead?
    
    Cheers, Phil