* [PATCH RFC 0/4] net: add bpfilter
@ 2018-02-16 13:40 Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 1/4] modules: allow insmod load regular elf binaries Daniel Borkmann
` (6 more replies)
0 siblings, 7 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-16 13:40 UTC (permalink / raw)
To: netdev; +Cc: netfilter-devel, davem, alexei.starovoitov, Daniel Borkmann
This is a very rough and early proof of concept that implements bpfilter.
The basic idea of bpfilter is that it can process iptables queries and
translate them in user space into BPF programs which can then get attached
at various locations. For simplicity, in this RFC we demo attaching them
to XDP layer, but any other location would work as well (e.g. at the tc
sch_clsact ingress/egress location or any other/new hook with equivalent
semantics).
Also, as a benefit from such design, we get BPF JIT compilation on x86_64,
arm64, ppc64, sparc64, mips64, s390x and arm32, but also rule offloading
into HW for free for Netronome NFP SmartNICs that are already capable of
offloading BPF since we can reuse all existing BPF infrastructure as the
back end. The user space iptables binary issuing rule addition or dumps was
left as-is, thus at some point any binaries against iptables uapi kernel
interface could transparently be supported in such manner in long term.
As rule translation can potentially become very complex, this is performed
entirely in user space. In order to ease deployment, request_module() code
is extended to allow user mode helpers to be invoked. Idea is that user mode
helpers are built as part of the kernel build and installed as traditional
kernel modules with .ko file extension into distro specified location,
such that from a distribution point of view, they are no different than
regular kernel modules. Thus, allow request_module() logic to load such
user mode helper (umh) binaries via:
request_module("foo") ->
call_umh("modprobe foo") ->
sys_finit_module(FD of /lib/modules/.../foo.ko) ->
call_umh(struct file)
Such approach enables kernel to delegate functionality traditionally done
by kernel modules into user space processes (either root or !root) and
reduces security attack surface of such new code, meaning in case of
potential bugs only the umh would crash but not the kernel. Another
advantage coming with that would be that bpfilter.ko can be debugged and
tested out of user space as well (e.g. opening the possibility to run
all clang sanitizers, fuzzers or test suites for checking translation).
Also, such architecture makes the kernel/user boundary very precise,
meaning requests can be handled and BPF translated in control plane part
in user space with its own user memory etc, while minimal data plane
bits are in kernel. It would also allow to remove old xtables modules
at some point from the kernel while keeping functionality in place.
In the implemented proof of concept we show that simple /32 src/dst IPs
are translated in such manner. More complex rules would be added later
as well, also different BPF code generation backends that can be selected
for the various attachment points, proper encoder/decoder for the uapi
requests, etc. This just starts out very simple and basic for the sake
of an early RFC to demo the idea.
In the below example, we show that dumping, loading and offloading of
one or multiple simple rules work, we show the bpftool XDP dump of the
generated BPF instruction sequence as well as a simple functional ping
test to enforce policy in such way.
Set rebased on top of 255442c93843 ("Merge tag 'docs-4.16' of [...]").
Feedback very welcome!
Various bpfilter usage examples from the PoC code:
1) Dumping current rules:
# iptables -t filter -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
2) ping test:
# ping -c 1 127.0.0.1 -I 127.0.0.2
PING 127.0.0.1 (127.0.0.1) from 127.0.0.2 : 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.040 ms
--- 127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.040/0.040/0.040/0.000 ms
3) Adding & dumping a simple rule:
# iptables -t filter -A INPUT -i lo -s 127.0.0.2/32 -d 127.0.0.1/32 -j DROP
# iptables -t filter -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all -- 127.0.0.2 localhost
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
4) Dump BPF generated code for that rule (on lo it's XDP generic, otherwise
native XDP for XDP supported drivers):
# bpftool p
18: xdp tag 6b07f663830d5b0c
loaded_at Feb 14/01:15 uid 0
xlated 208B not jited memlock 4096B
# bpftool p d x i 18
0: (bf) r9 = r1
1: (79) r2 = *(u64 *)(r9 +0)
2: (79) r3 = *(u64 *)(r9 +8)
3: (bf) r1 = r2
4: (07) r1 += 14
5: (bd) if r1 <= r3 goto pc+2
6: (b4) (u32) r0 = (u32) 2
7: (95) exit
8: (bf) r1 = r2
9: (b4) (u32) r5 = (u32) 0
10: (69) r4 = *(u16 *)(r1 +12)
11: (55) if r4 != 0x8 goto pc+9
12: (07) r1 += 34
13: (2d) if r1 > r3 goto pc+7
14: (07) r1 += -20
15: (61) r4 = *(u32 *)(r1 +12)
16: (55) if r4 != 0x200007f goto pc+1
17: (04) (u32) r5 += (u32) 1
18: (61) r4 = *(u32 *)(r1 +16)
19: (55) if r4 != 0x100007f goto pc+1
20: (04) (u32) r5 += (u32) 1
21: (55) if r5 != 0x2 goto pc+2
22: (b4) (u32) r0 = (u32) 1
23: (95) exit
24: (b4) (u32) r0 = (u32) 2
25: (95) exit
5) ping test:
# ping -c 1 127.0.0.1 -I 127.0.0.2
PING 127.0.0.1 (127.0.0.1) from 127.0.0.2 : 56(84) bytes of data.
--- 127.0.0.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
# ping -c 1 127.0.0.1 -I 127.0.0.1
PING 127.0.0.1 (127.0.0.1) from 127.0.0.1 : 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.018 ms
--- 127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.018/0.018/0.018/0.000 ms
# ping -c 1 127.0.0.2 -I 127.0.0.2
PING 127.0.0.2 (127.0.0.2) from 127.0.0.2 : 56(84) bytes of data.
64 bytes from 127.0.0.2: icmp_seq=1 ttl=64 time=0.018 ms
--- 127.0.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.018/0.018/0.018/0.000 ms
6) Adding & dumping a 2nd and 3rd rule:
# iptables -t filter -A INPUT -i lo -s 127.0.0.4/32 -d 127.0.0.3/32 -j DROP
# iptables -t filter -A INPUT -i lo -s 127.0.0.5/32 -j DROP
# iptables -t filter -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all -- 127.0.0.2 localhost
DROP all -- 127.0.0.4 127.0.0.3
DROP all -- anywhere 127.0.0.5
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
7) Dump BPF generated code again:
# bpftool p
20: xdp tag 19519bdd253cbfe5
loaded_at Feb 14/01:17 uid 0
xlated 440B not jited memlock 4096B
# bpftool p d x i 20
0: (bf) r9 = r1
1: (79) r2 = *(u64 *)(r9 +0)
2: (79) r3 = *(u64 *)(r9 +8)
3: (bf) r1 = r2
4: (07) r1 += 14
5: (bd) if r1 <= r3 goto pc+2
6: (b4) (u32) r0 = (u32) 2
7: (95) exit
8: (bf) r1 = r2
9: (b4) (u32) r5 = (u32) 0
10: (69) r4 = *(u16 *)(r1 +12)
11: (55) if r4 != 0x8 goto pc+9
12: (07) r1 += 34
13: (2d) if r1 > r3 goto pc+7
14: (07) r1 += -20
15: (61) r4 = *(u32 *)(r1 +12)
16: (55) if r4 != 0x200007f goto pc+1
17: (04) (u32) r5 += (u32) 1
18: (61) r4 = *(u32 *)(r1 +16)
19: (55) if r4 != 0x100007f goto pc+1
20: (04) (u32) r5 += (u32) 1
21: (55) if r5 != 0x2 goto pc+2
22: (b4) (u32) r0 = (u32) 1
23: (95) exit
24: (bf) r1 = r2
25: (b4) (u32) r5 = (u32) 0
26: (69) r4 = *(u16 *)(r1 +12)
27: (55) if r4 != 0x8 goto pc+9
28: (07) r1 += 34
29: (2d) if r1 > r3 goto pc+7
30: (07) r1 += -20
31: (61) r4 = *(u32 *)(r1 +12)
32: (55) if r4 != 0x400007f goto pc+1
33: (04) (u32) r5 += (u32) 1
34: (61) r4 = *(u32 *)(r1 +16)
35: (55) if r4 != 0x300007f goto pc+1
36: (04) (u32) r5 += (u32) 1
37: (55) if r5 != 0x2 goto pc+2
38: (b4) (u32) r0 = (u32) 1
39: (95) exit
40: (bf) r1 = r2
41: (b4) (u32) r5 = (u32) 0
42: (69) r4 = *(u16 *)(r1 +12)
43: (55) if r4 != 0x8 goto pc+6
44: (07) r1 += 34
45: (2d) if r1 > r3 goto pc+4
46: (07) r1 += -20
47: (61) r4 = *(u32 *)(r1 +12)
48: (55) if r4 != 0x500007f goto pc+1
49: (04) (u32) r5 += (u32) 1
50: (55) if r5 != 0x1 goto pc+2
51: (b4) (u32) r0 = (u32) 1
52: (95) exit
53: (b4) (u32) r0 = (u32) 2
54: (95) exit
8) ping test again:
# ping -c 1 127.0.0.4 -I 127.0.0.4
PING 127.0.0.4 (127.0.0.4) from 127.0.0.4 : 56(84) bytes of data.
64 bytes from 127.0.0.4: icmp_seq=1 ttl=64 time=0.032 ms
--- 127.0.0.4 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.032/0.032/0.032/0.000 ms
# ping -c 1 127.0.0.4 -I 127.0.0.3
PING 127.0.0.4 (127.0.0.4) from 127.0.0.3 : 56(84) bytes of data.
--- 127.0.0.4 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
# ping -c 1 127.0.0.1 -I 127.0.0.2
PING 127.0.0.1 (127.0.0.1) from 127.0.0.2 : 56(84) bytes of data.
--- 127.0.0.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
# ping -c 1 127.0.0.1 -I 127.0.0.5
PING 127.0.0.1 (127.0.0.1) from 127.0.0.5 : 56(84) bytes of data.
--- 127.0.0.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
9) Now example test with offload into nfp device:
# ethtool -i enp2s0
driver: nfp
version: 4.15.0+ SMP mod_unload
firmware-version: 0.0.5.5 0.17 bpf_xxxxxxx ebpf
expansion-rom-version:
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
# iptables -t filter -A INPUT -i enp2s0 -s 192.168.2.2/32 -j DROP
# bpftool p
1: xdp tag 88896d0ae0f463a6 dev enp2s0 ( <-- offloaded into HW )
loaded_at Feb 15/14:30 uid 0
xlated 184B jited 640B memlock 4096B
# bpftool p d x i 1
0: (bf) r9 = r1
1: (79) r2 = *(u64 *)(r9 +0)
2: (79) r3 = *(u64 *)(r9 +8)
3: (bf) r1 = r2
4: (07) r1 += 14
5: (bd) if r1 <= r3 goto pc+2
6: (b4) (u32) r0 = (u32) 2
7: (95) exit
8: (bf) r1 = r2
9: (b4) (u32) r5 = (u32) 0
10: (69) r4 = *(u16 *)(r1 +12)
11: (55) if r4 != 0x8 goto pc+6
12: (07) r1 += 34
13: (2d) if r1 > r3 goto pc+4
14: (07) r1 += -20
15: (61) r4 = *(u32 *)(r1 +12)
16: (55) if r4 != 0x202a8c0 goto pc+1
17: (04) (u32) r5 += (u32) 1
18: (55) if r5 != 0x1 goto pc+2
19: (b4) (u32) r0 = (u32) 1
20: (95) exit
21: (b4) (u32) r0 = (u32) 2
22: (95) exit
Thanks!
Alexei Starovoitov (2):
modules: allow insmod load regular elf binaries
bpf: introduce bpfilter commands
Daniel Borkmann (1):
bpf: rough bpfilter codegen example hack
David S. Miller (1):
net: initial bpfilter skeleton
fs/exec.c | 40 ++++-
include/linux/binfmts.h | 1 +
include/linux/bpfilter.h | 13 ++
include/linux/umh.h | 4 +
include/uapi/linux/bpf.h | 31 ++++
include/uapi/linux/bpfilter.h | 200 ++++++++++++++++++++++
kernel/bpf/syscall.c | 52 ++++++
kernel/module.c | 33 +++-
kernel/umh.c | 24 ++-
net/Kconfig | 2 +
net/Makefile | 1 +
net/bpfilter/Kconfig | 7 +
net/bpfilter/Makefile | 9 +
net/bpfilter/bpfilter.c | 106 ++++++++++++
net/bpfilter/bpfilter_mod.h | 373 ++++++++++++++++++++++++++++++++++++++++++
net/bpfilter/ctor.c | 91 +++++++++++
net/bpfilter/gen.c | 290 ++++++++++++++++++++++++++++++++
net/bpfilter/init.c | 36 ++++
net/bpfilter/sockopt.c | 236 ++++++++++++++++++++++++++
net/bpfilter/tables.c | 73 +++++++++
net/bpfilter/targets.c | 51 ++++++
net/bpfilter/tgts.c | 26 +++
net/ipv4/Makefile | 2 +
net/ipv4/bpfilter/Makefile | 2 +
net/ipv4/bpfilter/sockopt.c | 64 ++++++++
net/ipv4/ip_sockglue.c | 17 ++
26 files changed, 1767 insertions(+), 17 deletions(-)
create mode 100644 include/linux/bpfilter.h
create mode 100644 include/uapi/linux/bpfilter.h
create mode 100644 net/bpfilter/Kconfig
create mode 100644 net/bpfilter/Makefile
create mode 100644 net/bpfilter/bpfilter.c
create mode 100644 net/bpfilter/bpfilter_mod.h
create mode 100644 net/bpfilter/ctor.c
create mode 100644 net/bpfilter/gen.c
create mode 100644 net/bpfilter/init.c
create mode 100644 net/bpfilter/sockopt.c
create mode 100644 net/bpfilter/tables.c
create mode 100644 net/bpfilter/targets.c
create mode 100644 net/bpfilter/tgts.c
create mode 100644 net/ipv4/bpfilter/Makefile
create mode 100644 net/ipv4/bpfilter/sockopt.c
--
2.9.5
^ permalink raw reply [flat|nested] 73+ messages in thread
* [PATCH RFC 1/4] modules: allow insmod load regular elf binaries
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
@ 2018-02-16 13:40 ` Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 2/4] bpf: introduce bpfilter commands Daniel Borkmann
` (5 subsequent siblings)
6 siblings, 0 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-16 13:40 UTC (permalink / raw)
To: netdev; +Cc: netfilter-devel, davem, alexei.starovoitov, Alexei Starovoitov
From: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
fs/exec.c | 40 +++++++++++++++++++++++++++++++---------
include/linux/binfmts.h | 1 +
include/linux/umh.h | 4 ++++
kernel/module.c | 33 ++++++++++++++++++++++++++++-----
kernel/umh.c | 24 +++++++++++++++++++++---
5 files changed, 85 insertions(+), 17 deletions(-)
diff --git a/fs/exec.c b/fs/exec.c
index 7eb8d21..0483c43 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1691,14 +1691,13 @@ static int exec_binprm(struct linux_binprm *bprm)
/*
* sys_execve() executes a new program.
*/
-static int do_execveat_common(int fd, struct filename *filename,
- struct user_arg_ptr argv,
- struct user_arg_ptr envp,
- int flags)
+static int __do_execve_file(int fd, struct filename *filename,
+ struct user_arg_ptr argv,
+ struct user_arg_ptr envp,
+ int flags, struct file *file)
{
char *pathbuf = NULL;
struct linux_binprm *bprm;
- struct file *file;
struct files_struct *displaced;
int retval;
@@ -1737,7 +1736,8 @@ static int do_execveat_common(int fd, struct filename *filename,
check_unsafe_exec(bprm);
current->in_execve = 1;
- file = do_open_execat(fd, filename, flags);
+ if (!file)
+ file = do_open_execat(fd, filename, flags);
retval = PTR_ERR(file);
if (IS_ERR(file))
goto out_unmark;
@@ -1745,7 +1745,9 @@ static int do_execveat_common(int fd, struct filename *filename,
sched_exec();
bprm->file = file;
- if (fd == AT_FDCWD || filename->name[0] == '/') {
+ if (!filename) {
+ bprm->filename = "/dev/null";
+ } else if (fd == AT_FDCWD || filename->name[0] == '/') {
bprm->filename = filename->name;
} else {
if (filename->name[0] == '\0')
@@ -1811,7 +1813,8 @@ static int do_execveat_common(int fd, struct filename *filename,
task_numa_free(current);
free_bprm(bprm);
kfree(pathbuf);
- putname(filename);
+ if (filename)
+ putname(filename);
if (displaced)
put_files_struct(displaced);
return retval;
@@ -1834,10 +1837,29 @@ static int do_execveat_common(int fd, struct filename *filename,
if (displaced)
reset_files_struct(displaced);
out_ret:
- putname(filename);
+ if (filename)
+ putname(filename);
return retval;
}
+static int do_execveat_common(int fd, struct filename *filename,
+ struct user_arg_ptr argv,
+ struct user_arg_ptr envp,
+ int flags)
+{
+ struct file *file = NULL;
+
+ return __do_execve_file(fd, filename, argv, envp, flags, file);
+}
+
+int do_execve_file(struct file *file, void *__argv, void *__envp)
+{
+ struct user_arg_ptr argv = { .ptr.native = __argv };
+ struct user_arg_ptr envp = { .ptr.native = __envp };
+
+ return __do_execve_file(AT_FDCWD, NULL, argv, envp, 0, file);
+}
+
int do_execve(struct filename *filename,
const char __user *const __user *__argv,
const char __user *const __user *__envp)
diff --git a/include/linux/binfmts.h b/include/linux/binfmts.h
index b0abe21..c783a7b 100644
--- a/include/linux/binfmts.h
+++ b/include/linux/binfmts.h
@@ -147,5 +147,6 @@ extern int do_execveat(int, struct filename *,
const char __user * const __user *,
const char __user * const __user *,
int);
+int do_execve_file(struct file *file, void *__argv, void *__envp);
#endif /* _LINUX_BINFMTS_H */
diff --git a/include/linux/umh.h b/include/linux/umh.h
index 244aff6..68ddd4f 100644
--- a/include/linux/umh.h
+++ b/include/linux/umh.h
@@ -22,6 +22,7 @@ struct subprocess_info {
const char *path;
char **argv;
char **envp;
+ struct file *file;
int wait;
int retval;
int (*init)(struct subprocess_info *info, struct cred *new);
@@ -38,6 +39,9 @@ call_usermodehelper_setup(const char *path, char **argv, char **envp,
int (*init)(struct subprocess_info *info, struct cred *new),
void (*cleanup)(struct subprocess_info *), void *data);
+extern struct subprocess_info *
+call_usermodehelper_setup_file(struct file *file);
+
extern int
call_usermodehelper_exec(struct subprocess_info *info, int wait);
diff --git a/kernel/module.c b/kernel/module.c
index 1d65b2c..b0febe3 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -325,6 +325,7 @@ struct load_info {
struct {
unsigned int sym, str, mod, vers, info, pcpu;
} index;
+ struct file *file;
};
/*
@@ -2801,6 +2802,15 @@ static int module_sig_check(struct load_info *info, int flags)
}
#endif /* !CONFIG_MODULE_SIG */
+static int run_umh(struct file *file)
+{
+ struct subprocess_info *sub_info = call_usermodehelper_setup_file(file);
+
+ if (!file)
+ return -ENOMEM;
+ return call_usermodehelper_exec(sub_info, UMH_WAIT_EXEC);
+}
+
/* Sanity checks against invalid binaries, wrong arch, weird elf version. */
static int elf_header_check(struct load_info *info)
{
@@ -2808,7 +2818,6 @@ static int elf_header_check(struct load_info *info)
return -ENOEXEC;
if (memcmp(info->hdr->e_ident, ELFMAG, SELFMAG) != 0
- || info->hdr->e_type != ET_REL
|| !elf_check_arch(info->hdr)
|| info->hdr->e_shentsize != sizeof(Elf_Shdr))
return -ENOEXEC;
@@ -2818,6 +2827,11 @@ static int elf_header_check(struct load_info *info)
info->len - info->hdr->e_shoff))
return -ENOEXEC;
+ if (info->hdr->e_type == ET_EXEC)
+ return run_umh(info->file);
+
+ if (info->hdr->e_type != ET_REL)
+ return -ENOEXEC;
return 0;
}
@@ -3861,6 +3875,7 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
SYSCALL_DEFINE3(finit_module, int, fd, const char __user *, uargs, int, flags)
{
struct load_info info = { };
+ struct fd f;
loff_t size;
void *hdr;
int err;
@@ -3875,14 +3890,22 @@ SYSCALL_DEFINE3(finit_module, int, fd, const char __user *, uargs, int, flags)
|MODULE_INIT_IGNORE_VERMAGIC))
return -EINVAL;
- err = kernel_read_file_from_fd(fd, &hdr, &size, INT_MAX,
- READING_MODULE);
+ err = -EBADF;
+ f = fdget(fd);
+ if (!f.file)
+ goto out;
+
+ err = kernel_read_file(f.file, &hdr, &size, INT_MAX, READING_MODULE);
if (err)
- return err;
+ goto out;
info.hdr = hdr;
info.len = size;
+ info.file = f.file;
- return load_module(&info, uargs, flags);
+ err = load_module(&info, uargs, flags);
+out:
+ fdput(f);
+ return err;
}
static inline int within(unsigned long addr, void *start, unsigned long size)
diff --git a/kernel/umh.c b/kernel/umh.c
index 18e5fa4..073a686 100644
--- a/kernel/umh.c
+++ b/kernel/umh.c
@@ -97,9 +97,12 @@ static int call_usermodehelper_exec_async(void *data)
commit_creds(new);
- retval = do_execve(getname_kernel(sub_info->path),
- (const char __user *const __user *)sub_info->argv,
- (const char __user *const __user *)sub_info->envp);
+ if (sub_info->file)
+ retval = do_execve_file(sub_info->file, sub_info->argv, sub_info->envp);
+ else
+ retval = do_execve(getname_kernel(sub_info->path),
+ (const char __user *const __user *)sub_info->argv,
+ (const char __user *const __user *)sub_info->envp);
out:
sub_info->retval = retval;
/*
@@ -393,6 +396,21 @@ struct subprocess_info *call_usermodehelper_setup(const char *path, char **argv,
}
EXPORT_SYMBOL(call_usermodehelper_setup);
+struct subprocess_info *call_usermodehelper_setup_file(struct file *file)
+{
+ struct subprocess_info *sub_info;
+ sub_info = kzalloc(sizeof(struct subprocess_info), GFP_KERNEL);
+ if (!sub_info)
+ goto out;
+
+ INIT_WORK(&sub_info->work, call_usermodehelper_exec_work);
+
+ sub_info->path = "/dev/null";
+ sub_info->file = file;
+ out:
+ return sub_info;
+}
+
/**
* call_usermodehelper_exec - start a usermode application
* @sub_info: information about the subprocessa
--
2.9.5
^ permalink raw reply related [flat|nested] 73+ messages in thread
* [PATCH RFC 2/4] bpf: introduce bpfilter commands
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 1/4] modules: allow insmod load regular elf binaries Daniel Borkmann
@ 2018-02-16 13:40 ` Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 3/4] net: initial bpfilter skeleton Daniel Borkmann
` (4 subsequent siblings)
6 siblings, 0 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-16 13:40 UTC (permalink / raw)
To: netdev; +Cc: netfilter-devel, davem, alexei.starovoitov, Alexei Starovoitov
From: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/uapi/linux/bpf.h | 16 ++++++++++++++++
kernel/bpf/syscall.c | 41 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 57 insertions(+)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index db6bdc3..ea977e9 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -94,6 +94,8 @@ enum bpf_cmd {
BPF_MAP_GET_FD_BY_ID,
BPF_OBJ_GET_INFO_BY_FD,
BPF_PROG_QUERY,
+ BPFILTER_GET_CMD,
+ BPFILTER_REPLY,
};
enum bpf_map_type {
@@ -231,6 +233,17 @@ enum bpf_attach_type {
#define BPF_F_RDONLY (1U << 3)
#define BPF_F_WRONLY (1U << 4)
+struct bpfilter_get_cmd {
+ __u32 pid;
+ __u32 cmd;
+ __u64 addr;
+ __u32 len;
+};
+
+struct bpfilter_reply {
+ __u32 status;
+};
+
union bpf_attr {
struct { /* anonymous struct used by BPF_MAP_CREATE command */
__u32 map_type; /* one of enum bpf_map_type */
@@ -320,6 +333,9 @@ union bpf_attr {
__aligned_u64 prog_ids;
__u32 prog_cnt;
} query;
+
+ struct bpfilter_get_cmd bpfilter_get_cmd;
+ struct bpfilter_reply bpfilter_reply;
} __attribute__((aligned(8)));
/* BPF helper function descriptions:
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index e24aa32..e933bf9 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1840,6 +1840,41 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,
return err;
}
+DECLARE_WAIT_QUEUE_HEAD(bpfilter_get_cmd_wq);
+DECLARE_WAIT_QUEUE_HEAD(bpfilter_reply_wq);
+bool bpfilter_get_cmd_ready = false;
+bool bpfilter_reply_ready = false;
+struct bpfilter_get_cmd bpfilter_get_cmd_mbox;
+struct bpfilter_reply bpfilter_reply_mbox;
+
+#define BPFILTER_GET_CMD_LAST_FIELD bpfilter_get_cmd.len
+
+static int bpfilter_get_cmd(const union bpf_attr *attr,
+ union bpf_attr __user *uattr)
+{
+ if (CHECK_ATTR(BPFILTER_GET_CMD))
+ return -EINVAL;
+ wait_event_killable(bpfilter_get_cmd_wq, bpfilter_get_cmd_ready);
+ bpfilter_get_cmd_ready = false;
+ if (copy_to_user(&uattr->bpfilter_get_cmd, &bpfilter_get_cmd_mbox,
+ sizeof(bpfilter_get_cmd_mbox)))
+ return -EFAULT;
+ return 0;
+}
+
+#define BPFILTER_REPLY_LAST_FIELD bpfilter_reply.status
+
+static int bpfilter_reply(const union bpf_attr *attr,
+ union bpf_attr __user *uattr)
+{
+ if (CHECK_ATTR(BPFILTER_REPLY))
+ return -EINVAL;
+ bpfilter_reply_mbox.status = attr->bpfilter_reply.status;
+ bpfilter_reply_ready = true;
+ wake_up(&bpfilter_reply_wq);
+ return 0;
+}
+
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
{
union bpf_attr attr = {};
@@ -1917,6 +1952,12 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
case BPF_OBJ_GET_INFO_BY_FD:
err = bpf_obj_get_info_by_fd(&attr, uattr);
break;
+ case BPFILTER_GET_CMD:
+ err = bpfilter_get_cmd(&attr, uattr);
+ break;
+ case BPFILTER_REPLY:
+ err = bpfilter_reply(&attr, uattr);
+ break;
default:
err = -EINVAL;
break;
--
2.9.5
^ permalink raw reply related [flat|nested] 73+ messages in thread
* [PATCH RFC 3/4] net: initial bpfilter skeleton
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 1/4] modules: allow insmod load regular elf binaries Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 2/4] bpf: introduce bpfilter commands Daniel Borkmann
@ 2018-02-16 13:40 ` Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 4/4] bpf: rough bpfilter codegen example hack Daniel Borkmann
` (3 subsequent siblings)
6 siblings, 0 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-16 13:40 UTC (permalink / raw)
To: netdev; +Cc: netfilter-devel, davem, alexei.starovoitov, Alexei Starovoitov
From: "David S. Miller" <davem@davemloft.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpfilter.h | 13 +++
include/uapi/linux/bpfilter.h | 200 ++++++++++++++++++++++++++++++++++++++++++
net/Kconfig | 2 +
net/Makefile | 1 +
net/bpfilter/Kconfig | 7 ++
net/bpfilter/Makefile | 9 ++
net/bpfilter/bpfilter.c | 89 +++++++++++++++++++
net/bpfilter/bpfilter_mod.h | 96 ++++++++++++++++++++
net/bpfilter/ctor.c | 80 +++++++++++++++++
net/bpfilter/init.c | 33 +++++++
net/bpfilter/sockopt.c | 153 ++++++++++++++++++++++++++++++++
net/bpfilter/tables.c | 70 +++++++++++++++
net/bpfilter/targets.c | 51 +++++++++++
net/bpfilter/tgts.c | 25 ++++++
net/ipv4/Makefile | 2 +
net/ipv4/bpfilter/Makefile | 2 +
net/ipv4/bpfilter/sockopt.c | 49 +++++++++++
net/ipv4/ip_sockglue.c | 17 ++++
18 files changed, 899 insertions(+)
create mode 100644 include/linux/bpfilter.h
create mode 100644 include/uapi/linux/bpfilter.h
create mode 100644 net/bpfilter/Kconfig
create mode 100644 net/bpfilter/Makefile
create mode 100644 net/bpfilter/bpfilter.c
create mode 100644 net/bpfilter/bpfilter_mod.h
create mode 100644 net/bpfilter/ctor.c
create mode 100644 net/bpfilter/init.c
create mode 100644 net/bpfilter/sockopt.c
create mode 100644 net/bpfilter/tables.c
create mode 100644 net/bpfilter/targets.c
create mode 100644 net/bpfilter/tgts.c
create mode 100644 net/ipv4/bpfilter/Makefile
create mode 100644 net/ipv4/bpfilter/sockopt.c
diff --git a/include/linux/bpfilter.h b/include/linux/bpfilter.h
new file mode 100644
index 0000000..26adad1
--- /dev/null
+++ b/include/linux/bpfilter.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_BPFILTER_H
+#define _LINUX_BPFILTER_H
+
+#include <uapi/linux/bpfilter.h>
+
+struct sock;
+int bpfilter_ip_set_sockopt(struct sock *sk, int optname, char *optval,
+ unsigned int optlen);
+int bpfilter_ip_get_sockopt(struct sock *sk, int optname, char *optval,
+ int *optlen);
+#endif
+
diff --git a/include/uapi/linux/bpfilter.h b/include/uapi/linux/bpfilter.h
new file mode 100644
index 0000000..38d54e9
--- /dev/null
+++ b/include/uapi/linux/bpfilter.h
@@ -0,0 +1,200 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _UAPI_LINUX_BPFILTER_H
+#define _UAPI_LINUX_BPFILTER_H
+
+#include <linux/if.h>
+
+enum {
+ BPFILTER_IPT_SO_SET_REPLACE = 64,
+ BPFILTER_IPT_SO_SET_ADD_COUNTERS = 65,
+ BPFILTER_IPT_SET_MAX,
+};
+
+enum {
+ BPFILTER_IPT_SO_GET_INFO = 64,
+ BPFILTER_IPT_SO_GET_ENTRIES = 65,
+ BPFILTER_IPT_SO_GET_REVISION_MATCH = 66,
+ BPFILTER_IPT_SO_GET_REVISION_TARGET = 67,
+ BPFILTER_IPT_GET_MAX,
+};
+
+enum {
+ BPFILTER_XT_TABLE_MAXNAMELEN = 32,
+};
+
+enum {
+ BPFILTER_NF_DROP = 0,
+ BPFILTER_NF_ACCEPT = 1,
+ BPFILTER_NF_STOLEN = 2,
+ BPFILTER_NF_QUEUE = 3,
+ BPFILTER_NF_REPEAT = 4,
+ BPFILTER_NF_STOP = 5,
+ BPFILTER_NF_MAX_VERDICT = BPFILTER_NF_STOP,
+};
+
+enum {
+ BPFILTER_INET_HOOK_PRE_ROUTING = 0,
+ BPFILTER_INET_HOOK_LOCAL_IN = 1,
+ BPFILTER_INET_HOOK_FORWARD = 2,
+ BPFILTER_INET_HOOK_LOCAL_OUT = 3,
+ BPFILTER_INET_HOOK_POST_ROUTING = 4,
+ BPFILTER_INET_HOOK_MAX,
+};
+
+enum {
+ BPFILTER_PROTO_UNSPEC = 0,
+ BPFILTER_PROTO_INET = 1,
+ BPFILTER_PROTO_IPV4 = 2,
+ BPFILTER_PROTO_ARP = 3,
+ BPFILTER_PROTO_NETDEV = 5,
+ BPFILTER_PROTO_BRIDGE = 7,
+ BPFILTER_PROTO_IPV6 = 10,
+ BPFILTER_PROTO_DECNET = 12,
+ BPFILTER_PROTO_NUMPROTO,
+};
+
+#ifndef INT_MAX
+#define INT_MAX ((int)(~0U>>1))
+#endif
+#ifndef INT_MIN
+#define INT_MIN (-INT_MAX - 1)
+#endif
+
+enum {
+ BPFILTER_IP_PRI_FIRST = INT_MIN,
+ BPFILTER_IP_PRI_CONNTRACK_DEFRAG = -400,
+ BPFILTER_IP_PRI_RAW = -300,
+ BPFILTER_IP_PRI_SELINUX_FIRST = -225,
+ BPFILTER_IP_PRI_CONNTRACK = -200,
+ BPFILTER_IP_PRI_MANGLE = -150,
+ BPFILTER_IP_PRI_NAT_DST = -100,
+ BPFILTER_IP_PRI_FILTER = 0,
+ BPFILTER_IP_PRI_SECURITY = 50,
+ BPFILTER_IP_PRI_NAT_SRC = 100,
+ BPFILTER_IP_PRI_SELINUX_LAST = 225,
+ BPFILTER_IP_PRI_CONNTRACK_HELPER = 300,
+ BPFILTER_IP_PRI_CONNTRACK_CONFIRM = INT_MAX,
+ BPFILTER_IP_PRI_LAST = INT_MAX,
+};
+
+#define BPFILTER_FUNCTION_MAXNAMELEN 30
+#define BPFILTER_EXTENSION_MAXNAMELEN 29
+#define BPFILTER_TABLE_MAXNAMELEN 32
+
+struct bpfilter_match;
+struct bpfilter_entry_match {
+ union {
+ struct {
+ __u16 match_size;
+ char name[BPFILTER_EXTENSION_MAXNAMELEN];
+ __u8 revision;
+ } user;
+ struct {
+ __u16 match_size;
+ struct bpfilter_match *match;
+ } kernel;
+ __u16 match_size;
+ } u;
+ unsigned char data[0];
+};
+
+struct bpfilter_target;
+struct bpfilter_entry_target {
+ union {
+ struct {
+ __u16 target_size;
+ char name[BPFILTER_EXTENSION_MAXNAMELEN];
+ __u8 revision;
+ } user;
+ struct {
+ __u16 target_size;
+ struct bpfilter_target *target;
+ } kernel;
+ __u16 target_size;
+ } u;
+ unsigned char data[0];
+};
+
+struct bpfilter_standard_target {
+ struct bpfilter_entry_target target;
+ int verdict;
+};
+
+struct bpfilter_error_target {
+ struct bpfilter_entry_target target;
+ char error_name[BPFILTER_FUNCTION_MAXNAMELEN];
+};
+
+#define __ALIGN_KERNEL(x, a) __ALIGN_KERNEL_MASK(x, (typeof(x))(a) - 1)
+#define __ALIGN_KERNEL_MASK(x, mask) (((x) + (mask)) & ~(mask))
+
+#define BPFILTER_ALIGN(__X) \
+ __ALIGN_KERNEL(__X, __alignof__(__u64))
+
+#define BPFILTER_TARGET_INIT(__name, __size) \
+{ \
+ .target.u.user = { \
+ .target_size = BPFILTER_ALIGN(__size), \
+ .name = (__name), \
+ }, \
+}
+#define BPFILTER_STANDARD_TARGET ""
+#define BPFILTER_ERROR_TARGET "ERROR"
+
+struct bpfilter_xt_counters {
+ __u64 packet_cnt;
+ __u64 byte_cnt;
+};
+
+struct bpfilter_ipt_ip {
+ __u32 src;
+ __u32 dst;
+ __u32 src_mask;
+ __u32 dst_mask;
+ char in_iface[IFNAMSIZ];
+ char out_iface[IFNAMSIZ];
+ __u8 in_iface_mask[IFNAMSIZ];
+ __u8 out_iface_mask[IFNAMSIZ];
+ __u16 protocol;
+ __u8 flags;
+ __u8 inv_flags;
+};
+
+struct bpfilter_ipt_entry {
+ struct bpfilter_ipt_ip ip;
+ __u32 bfcache;
+ __u16 target_offset;
+ __u16 next_offset;
+ __u32 camefrom;
+ struct bpfilter_xt_counters cntrs;
+ __u8 elems[0];
+};
+
+struct bpfilter_ipt_get_info {
+ char name[BPFILTER_XT_TABLE_MAXNAMELEN];
+ __u32 valid_hooks;
+ __u32 hook_entry[BPFILTER_INET_HOOK_MAX];
+ __u32 underflow[BPFILTER_INET_HOOK_MAX];
+ __u32 num_entries;
+ __u32 size;
+};
+
+struct bpfilter_ipt_get_entries {
+ char name[BPFILTER_XT_TABLE_MAXNAMELEN];
+ __u32 size;
+ struct bpfilter_ipt_entry entries[0];
+};
+
+struct bpfilter_ipt_replace {
+ char name[BPFILTER_XT_TABLE_MAXNAMELEN];
+ __u32 valid_hooks;
+ __u32 num_entries;
+ __u32 size;
+ __u32 hook_entry[BPFILTER_INET_HOOK_MAX];
+ __u32 underflow[BPFILTER_INET_HOOK_MAX];
+ __u32 num_counters;
+ struct bpfilter_xt_counters *cntrs;
+ struct bpfilter_ipt_entry entries[0];
+};
+
+#endif /* _UAPI_LINUX_BPFILTER_H */
diff --git a/net/Kconfig b/net/Kconfig
index 37ec8e6..ec96506 100644
--- a/net/Kconfig
+++ b/net/Kconfig
@@ -201,6 +201,8 @@ source "net/bridge/netfilter/Kconfig"
endif
+source "net/bpfilter/Kconfig"
+
source "net/dccp/Kconfig"
source "net/sctp/Kconfig"
source "net/rds/Kconfig"
diff --git a/net/Makefile b/net/Makefile
index 14fede5..c388b3d 100644
--- a/net/Makefile
+++ b/net/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_TLS) += tls/
obj-$(CONFIG_XFRM) += xfrm/
obj-$(CONFIG_UNIX) += unix/
obj-$(CONFIG_NET) += ipv6/
+obj-$(CONFIG_BPFILTER) += bpfilter/
obj-$(CONFIG_PACKET) += packet/
obj-$(CONFIG_NET_KEY) += key/
obj-$(CONFIG_BRIDGE) += bridge/
diff --git a/net/bpfilter/Kconfig b/net/bpfilter/Kconfig
new file mode 100644
index 0000000..d29b5cb
--- /dev/null
+++ b/net/bpfilter/Kconfig
@@ -0,0 +1,7 @@
+menuconfig BPFILTER
+ bool "BPF Filter Configuration"
+ depends on NET && BPF
+
+if BPFILTER
+
+endif
diff --git a/net/bpfilter/Makefile b/net/bpfilter/Makefile
new file mode 100644
index 0000000..5e05505
--- /dev/null
+++ b/net/bpfilter/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Makefile for the Linux BPFILTER layer.
+#
+
+hostprogs-y := bpfilter.ko
+always := $(hostprogs-y)
+bpfilter.ko-objs := bpfilter.o tgts.o targets.o tables.o init.o ctor.o sockopt.o
+HOSTCFLAGS += -I. -Itools/include/
diff --git a/net/bpfilter/bpfilter.c b/net/bpfilter/bpfilter.c
new file mode 100644
index 0000000..445ae65
--- /dev/null
+++ b/net/bpfilter/bpfilter.c
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <sys/uio.h>
+#include <errno.h>
+#include <stdio.h>
+#include <sys/socket.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include "include/uapi/linux/bpf.h"
+#include <asm/unistd.h>
+#include "bpfilter_mod.h"
+
+extern long int syscall (long int __sysno, ...);
+
+static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
+ unsigned int size)
+{
+ return syscall(321, cmd, attr, size);
+}
+
+int pid;
+int debug_fd;
+
+int copy_from_user(void *dst, void *addr, int len)
+{
+ struct iovec local;
+ struct iovec remote;
+
+ local.iov_base = dst;
+ local.iov_len = len;
+ remote.iov_base = addr;
+ remote.iov_len = len;
+ return process_vm_readv(pid, &local, 1, &remote, 1, 0) != len;
+}
+
+int copy_to_user(void *addr, const void *src, int len)
+{
+ struct iovec local;
+ struct iovec remote;
+
+ local.iov_base = (void *) src;
+ local.iov_len = len;
+ remote.iov_base = addr;
+ remote.iov_len = len;
+ return process_vm_writev(pid, &local, 1, &remote, 1, 0) != len;
+}
+
+static int handle_cmd(struct bpfilter_get_cmd *cmd)
+{
+ pid = cmd->pid;
+ switch (cmd->cmd) {
+ case BPFILTER_IPT_SO_GET_INFO:
+ return bpfilter_get_info((void *) (long) cmd->addr, cmd->len);
+ case BPFILTER_IPT_SO_GET_ENTRIES:
+ return bpfilter_get_entries((void *) (long) cmd->addr, cmd->len);
+ default:
+ break;
+ }
+ return -ENOPROTOOPT;
+}
+
+static void loop(void)
+{
+ bpfilter_tables_init();
+ bpfilter_ipv4_init();
+
+ while (1) {
+ union bpf_attr get_cmd = {};
+ union bpf_attr reply = {};
+ struct bpfilter_get_cmd *cmd;
+
+ sys_bpf(BPFILTER_GET_CMD, &get_cmd, sizeof(get_cmd));
+ cmd = &get_cmd.bpfilter_get_cmd;
+
+ dprintf(debug_fd, "pid %d cmd %d addr %llx len %d\n",
+ cmd->pid, cmd->cmd, cmd->addr, cmd->len);
+
+ reply.bpfilter_reply.status = handle_cmd(cmd);
+ sys_bpf(BPFILTER_REPLY, &reply, sizeof(reply));
+ }
+}
+
+int main(void)
+{
+ debug_fd = open("/tmp/aa", 00000002 | 00000100);
+ loop();
+ close(debug_fd);
+ return 0;
+}
diff --git a/net/bpfilter/bpfilter_mod.h b/net/bpfilter/bpfilter_mod.h
new file mode 100644
index 0000000..f0de41b
--- /dev/null
+++ b/net/bpfilter/bpfilter_mod.h
@@ -0,0 +1,96 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_BPFILTER_INTERNAL_H
+#define _LINUX_BPFILTER_INTERNAL_H
+
+#include "include/uapi/linux/bpfilter.h"
+#include <linux/list.h>
+
+struct bpfilter_table {
+ struct hlist_node hash;
+ u32 valid_hooks;
+ struct bpfilter_table_info *info;
+ int hold;
+ u8 family;
+ int priority;
+ const char name[BPFILTER_XT_TABLE_MAXNAMELEN];
+};
+
+struct bpfilter_table_info {
+ unsigned int size;
+ u32 num_entries;
+ unsigned int initial_entries;
+ unsigned int hook_entry[BPFILTER_INET_HOOK_MAX];
+ unsigned int underflow[BPFILTER_INET_HOOK_MAX];
+ unsigned int stacksize;
+ void ***jumpstack;
+ unsigned char entries[0] __aligned(8);
+};
+
+struct bpfilter_table *bpfilter_table_get_by_name(const char *name, int name_len);
+void bpfilter_table_put(struct bpfilter_table *tbl);
+int bpfilter_table_add(struct bpfilter_table *tbl);
+
+struct bpfilter_ipt_standard {
+ struct bpfilter_ipt_entry entry;
+ struct bpfilter_standard_target target;
+};
+
+struct bpfilter_ipt_error {
+ struct bpfilter_ipt_entry entry;
+ struct bpfilter_error_target target;
+};
+
+#define BPFILTER_IPT_ENTRY_INIT(__sz) \
+{ \
+ .target_offset = sizeof(struct bpfilter_ipt_entry), \
+ .next_offset = (__sz), \
+}
+
+#define BPFILTER_IPT_STANDARD_INIT(__verdict) \
+{ \
+ .entry = BPFILTER_IPT_ENTRY_INIT(sizeof(struct bpfilter_ipt_standard)), \
+ .target = BPFILTER_TARGET_INIT(BPFILTER_STANDARD_TARGET, \
+ sizeof(struct bpfilter_standard_target)),\
+ .target.verdict = -(__verdict) - 1, \
+}
+
+#define BPFILTER_IPT_ERROR_INIT \
+{ \
+ .entry = BPFILTER_IPT_ENTRY_INIT(sizeof(struct bpfilter_ipt_error)), \
+ .target = BPFILTER_TARGET_INIT(BPFILTER_ERROR_TARGET, \
+ sizeof(struct bpfilter_error_target)), \
+ .target.error_name = "ERROR", \
+}
+
+struct bpfilter_target {
+ struct list_head all_target_list;
+ const char name[BPFILTER_EXTENSION_MAXNAMELEN];
+ unsigned int size;
+ int hold;
+ u16 family;
+ u8 rev;
+};
+
+struct bpfilter_target *bpfilter_target_get_by_name(const char *name);
+void bpfilter_target_put(struct bpfilter_target *tgt);
+int bpfilter_target_add(struct bpfilter_target *tgt);
+
+struct bpfilter_table_info *bpfilter_ipv4_table_ctor(struct bpfilter_table *tbl);
+int bpfilter_ipv4_register_targets(void);
+void bpfilter_tables_init(void);
+int bpfilter_get_info(void *addr, int len);
+int bpfilter_get_entries(void *cmd, int len);
+int bpfilter_ipv4_init(void);
+
+int copy_from_user(void *dst, void *addr, int len);
+int copy_to_user(void *addr, const void *src, int len);
+#define put_user(x, ptr) \
+({ \
+ __typeof__(*(ptr)) __x = (x); \
+ copy_to_user(ptr, &__x, sizeof(*(ptr))); \
+})
+extern int pid;
+extern int debug_fd;
+#define ENOTSUPP 524
+
+#endif
diff --git a/net/bpfilter/ctor.c b/net/bpfilter/ctor.c
new file mode 100644
index 0000000..efb7fee
--- /dev/null
+++ b/net/bpfilter/ctor.c
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <sys/socket.h>
+#include <linux/bitops.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include "bpfilter_mod.h"
+
+unsigned int __sw_hweight32(unsigned int w)
+{
+ w -= (w >> 1) & 0x55555555;
+ w = (w & 0x33333333) + ((w >> 2) & 0x33333333);
+ w = (w + (w >> 4)) & 0x0f0f0f0f;
+ return (w * 0x01010101) >> 24;
+}
+
+struct bpfilter_table_info *bpfilter_ipv4_table_ctor(struct bpfilter_table *tbl)
+{
+ unsigned int num_hooks = hweight32(tbl->valid_hooks);
+ struct bpfilter_ipt_standard *tgts;
+ struct bpfilter_table_info *info;
+ struct bpfilter_ipt_error *term;
+ unsigned int mask, offset, h, i;
+ unsigned int size, alloc_size;
+
+ size = sizeof(struct bpfilter_ipt_standard) * num_hooks;
+ size += sizeof(struct bpfilter_ipt_error);
+
+ alloc_size = size + sizeof(struct bpfilter_table_info);
+
+ info = malloc(alloc_size);
+ if (!info)
+ return NULL;
+
+ info->num_entries = num_hooks + 1;
+ info->size = size;
+
+ tgts = (struct bpfilter_ipt_standard *) (info + 1);
+ term = (struct bpfilter_ipt_error *) (tgts + num_hooks);
+
+ mask = tbl->valid_hooks;
+ offset = 0;
+ h = 0;
+ i = 0;
+ dprintf(debug_fd, "mask %x num_hooks %d\n", mask, num_hooks);
+ while (mask) {
+ struct bpfilter_ipt_standard *t;
+
+ if (!(mask & 1))
+ goto next;
+
+ info->hook_entry[h] = offset;
+ info->underflow[h] = offset;
+ t = &tgts[i++];
+ *t = (struct bpfilter_ipt_standard)
+ BPFILTER_IPT_STANDARD_INIT(BPFILTER_NF_ACCEPT);
+ t->target.target.u.kernel.target =
+ bpfilter_target_get_by_name(t->target.target.u.user.name);
+ dprintf(debug_fd, "user.name %s\n", t->target.target.u.user.name);
+ if (!t->target.target.u.kernel.target)
+ goto out_fail;
+
+ offset += sizeof(struct bpfilter_ipt_standard);
+ next:
+ mask >>= 1;
+ h++;
+ }
+ *term = (struct bpfilter_ipt_error) BPFILTER_IPT_ERROR_INIT;
+ term->target.target.u.kernel.target =
+ bpfilter_target_get_by_name(term->target.target.u.user.name);
+ dprintf(debug_fd, "user.name %s\n", term->target.target.u.user.name);
+ if (!term->target.target.u.kernel.target)
+ goto out_fail;
+
+ dprintf(debug_fd, "info %p\n", info);
+ return info;
+
+out_fail:
+ free(info);
+ return NULL;
+}
diff --git a/net/bpfilter/init.c b/net/bpfilter/init.c
new file mode 100644
index 0000000..699f3f6
--- /dev/null
+++ b/net/bpfilter/init.c
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <sys/socket.h>
+#include <errno.h>
+#include "bpfilter_mod.h"
+
+static struct bpfilter_table filter_table_ipv4 = {
+ .name = "filter",
+ .valid_hooks = ((1<<BPFILTER_INET_HOOK_LOCAL_IN) |
+ (1<<BPFILTER_INET_HOOK_FORWARD) |
+ (1<<BPFILTER_INET_HOOK_LOCAL_OUT)),
+ .family = BPFILTER_PROTO_IPV4,
+ .priority = BPFILTER_IP_PRI_FILTER,
+};
+
+int bpfilter_ipv4_init(void)
+{
+ struct bpfilter_table *t = &filter_table_ipv4;
+ struct bpfilter_table_info *info;
+ int err;
+
+ err = bpfilter_ipv4_register_targets();
+ if (err)
+ return err;
+
+ info = bpfilter_ipv4_table_ctor(t);
+ if (!info)
+ return -ENOMEM;
+
+ t->info = info;
+
+ return bpfilter_table_add(&filter_table_ipv4);
+}
+
diff --git a/net/bpfilter/sockopt.c b/net/bpfilter/sockopt.c
new file mode 100644
index 0000000..43687da
--- /dev/null
+++ b/net/bpfilter/sockopt.c
@@ -0,0 +1,153 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <sys/socket.h>
+#include <errno.h>
+#include <string.h>
+#include <stdio.h>
+#include "bpfilter_mod.h"
+
+static int fetch_name(void *addr, int len, char *name, int name_len)
+{
+ if (copy_from_user(name, addr, name_len))
+ return -EFAULT;
+
+ name[BPFILTER_XT_TABLE_MAXNAMELEN-1] = '\0';
+ return 0;
+}
+
+int bpfilter_get_info(void *addr, int len)
+{
+ char name[BPFILTER_XT_TABLE_MAXNAMELEN];
+ struct bpfilter_ipt_get_info resp;
+ struct bpfilter_table_info *info;
+ struct bpfilter_table *tbl;
+ int err;
+
+ if (len != sizeof(struct bpfilter_ipt_get_info))
+ return -EINVAL;
+
+ err = fetch_name(addr, len, name, sizeof(name));
+ if (err)
+ return err;
+
+ tbl = bpfilter_table_get_by_name(name, strlen(name));
+ if (!tbl)
+ return -ENOENT;
+
+ info = tbl->info;
+ if (!info) {
+ err = -ENOENT;
+ goto out_put;
+ }
+
+ memset(&resp, 0, sizeof(resp));
+ memcpy(resp.name, name, sizeof(resp.name));
+ resp.valid_hooks = tbl->valid_hooks;
+ memcpy(&resp.hook_entry, info->hook_entry, sizeof(resp.hook_entry));
+ memcpy(&resp.underflow, info->underflow, sizeof(resp.underflow));
+ resp.num_entries = info->num_entries;
+ resp.size = info->size;
+
+ err = 0;
+ if (copy_to_user(addr, &resp, len))
+ err = -EFAULT;
+out_put:
+ bpfilter_table_put(tbl);
+ return err;
+}
+
+static int copy_target(struct bpfilter_standard_target *ut,
+ struct bpfilter_standard_target *kt)
+{
+ struct bpfilter_target *tgt;
+ int sz;
+
+
+ if (put_user(kt->target.u.target_size,
+ &ut->target.u.target_size))
+ return -EFAULT;
+
+ tgt = kt->target.u.kernel.target;
+ if (copy_to_user(ut->target.u.user.name, tgt->name, strlen(tgt->name)))
+ return -EFAULT;
+
+ if (put_user(tgt->rev, &ut->target.u.user.revision))
+ return -EFAULT;
+
+ sz = tgt->size;
+ if (copy_to_user(ut->target.data, kt->target.data, sz))
+ return -EFAULT;
+
+ return 0;
+}
+
+static int do_get_entries(void *up,
+ struct bpfilter_table *tbl,
+ struct bpfilter_table_info *info)
+{
+ unsigned int total_size = info->size;
+ const struct bpfilter_ipt_entry *ent;
+ unsigned int off;
+ void *base;
+
+ base = info->entries;
+
+ for (off = 0; off < total_size; off += ent->next_offset) {
+ struct bpfilter_xt_counters *cntrs;
+ struct bpfilter_standard_target *tgt;
+
+ ent = base + off;
+ if (copy_to_user(up + off, ent, sizeof(*ent)))
+ return -EFAULT;
+
+ /* XXX Just clear counters for now. XXX */
+ cntrs = up + off + offsetof(struct bpfilter_ipt_entry, cntrs);
+ if (put_user(0, &cntrs->packet_cnt) ||
+ put_user(0, &cntrs->byte_cnt))
+ return -EINVAL;
+
+ tgt = (void *) ent + ent->target_offset;
+ dprintf(debug_fd, "target.verdict %d\n", tgt->verdict);
+ if (copy_target(up + off + ent->target_offset, tgt))
+ return -EFAULT;
+ }
+ return 0;
+}
+
+int bpfilter_get_entries(void *cmd, int len)
+{
+ struct bpfilter_ipt_get_entries *uptr = cmd;
+ struct bpfilter_ipt_get_entries req;
+ struct bpfilter_table_info *info;
+ struct bpfilter_table *tbl;
+ int err;
+
+ if (len < sizeof(struct bpfilter_ipt_get_entries))
+ return -EINVAL;
+
+ if (copy_from_user(&req, cmd, sizeof(req)))
+ return -EFAULT;
+
+ tbl = bpfilter_table_get_by_name(req.name, strlen(req.name));
+ if (!tbl)
+ return -ENOENT;
+
+ info = tbl->info;
+ if (!info) {
+ err = -ENOENT;
+ goto out_put;
+ }
+
+ if (info->size != req.size) {
+ err = -EINVAL;
+ goto out_put;
+ }
+
+ err = do_get_entries(uptr->entries, tbl, info);
+ dprintf(debug_fd, "do_get_entries %d req.size %d\n", err, req.size);
+
+out_put:
+ bpfilter_table_put(tbl);
+
+ return err;
+}
+
diff --git a/net/bpfilter/tables.c b/net/bpfilter/tables.c
new file mode 100644
index 0000000..9a96599
--- /dev/null
+++ b/net/bpfilter/tables.c
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <sys/socket.h>
+#include <errno.h>
+#include <string.h>
+#include <linux/hashtable.h>
+#include "bpfilter_mod.h"
+
+static unsigned int full_name_hash(const void *salt, const char *name, unsigned int len)
+{
+ unsigned int hash = 0;
+ int i;
+
+ for (i = 0; i < len; i++)
+ hash ^= *(name + i);
+ return hash;
+}
+
+DEFINE_HASHTABLE(bpfilter_tables, 4);
+//DEFINE_MUTEX(bpfilter_table_mutex);
+
+struct bpfilter_table *bpfilter_table_get_by_name(const char *name, int name_len)
+{
+ unsigned int hval = full_name_hash(NULL, name, name_len);
+ struct bpfilter_table *tbl;
+
+// mutex_lock(&bpfilter_table_mutex);
+ hash_for_each_possible(bpfilter_tables, tbl, hash, hval) {
+ if (!strcmp(name, tbl->name)) {
+ tbl->hold++;
+ goto out;
+ }
+ }
+ tbl = NULL;
+out:
+// mutex_unlock(&bpfilter_table_mutex);
+ return tbl;
+}
+
+void bpfilter_table_put(struct bpfilter_table *tbl)
+{
+// mutex_lock(&bpfilter_table_mutex);
+ tbl->hold--;
+// mutex_unlock(&bpfilter_table_mutex);
+}
+
+int bpfilter_table_add(struct bpfilter_table *tbl)
+{
+ unsigned int hval = full_name_hash(NULL, tbl->name, strlen(tbl->name));
+ struct bpfilter_table *srch;
+
+// mutex_lock(&bpfilter_table_mutex);
+ hash_for_each_possible(bpfilter_tables, srch, hash, hval) {
+ if (!strcmp(srch->name, tbl->name))
+ goto exists;
+ }
+ hash_add(bpfilter_tables, &tbl->hash, hval);
+// mutex_unlock(&bpfilter_table_mutex);
+
+ return 0;
+
+exists:
+// mutex_unlock(&bpfilter_table_mutex);
+ return -EEXIST;
+}
+
+void bpfilter_tables_init(void)
+{
+ hash_init(bpfilter_tables);
+}
+
diff --git a/net/bpfilter/targets.c b/net/bpfilter/targets.c
new file mode 100644
index 0000000..4086ac8
--- /dev/null
+++ b/net/bpfilter/targets.c
@@ -0,0 +1,51 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <sys/socket.h>
+#include <errno.h>
+#include <string.h>
+#include "bpfilter_mod.h"
+
+//DEFINE_MUTEX(bpfilter_target_mutex);
+static LIST_HEAD(bpfilter_targets);
+
+struct bpfilter_target *bpfilter_target_get_by_name(const char *name)
+{
+ struct bpfilter_target *tgt;
+
+// mutex_lock(&bpfilter_target_mutex);
+ list_for_each_entry(tgt, &bpfilter_targets, all_target_list) {
+ if (!strcmp(tgt->name, name)) {
+ tgt->hold++;
+ goto out;
+ }
+ }
+ tgt = NULL;
+out:
+// mutex_unlock(&bpfilter_target_mutex);
+ return tgt;
+}
+
+void bpfilter_target_put(struct bpfilter_target *tgt)
+{
+// mutex_lock(&bpfilter_target_mutex);
+ tgt->hold--;
+// mutex_unlock(&bpfilter_target_mutex);
+}
+
+int bpfilter_target_add(struct bpfilter_target *tgt)
+{
+ struct bpfilter_target *srch;
+
+// mutex_lock(&bpfilter_target_mutex);
+ list_for_each_entry(srch, &bpfilter_targets, all_target_list) {
+ if (!strcmp(srch->name, tgt->name))
+ goto exists;
+ }
+ list_add_tail(&tgt->all_target_list, &bpfilter_targets);
+// mutex_unlock(&bpfilter_target_mutex);
+ return 0;
+
+exists:
+// mutex_unlock(&bpfilter_target_mutex);
+ return -EEXIST;
+}
+
diff --git a/net/bpfilter/tgts.c b/net/bpfilter/tgts.c
new file mode 100644
index 0000000..eac5e8a
--- /dev/null
+++ b/net/bpfilter/tgts.c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <sys/socket.h>
+#include "bpfilter_mod.h"
+
+struct bpfilter_target std_tgt = {
+ .name = BPFILTER_STANDARD_TARGET,
+ .family = BPFILTER_PROTO_IPV4,
+ .size = sizeof(int),
+};
+
+struct bpfilter_target err_tgt = {
+ .name = BPFILTER_ERROR_TARGET,
+ .family = BPFILTER_PROTO_IPV4,
+ .size = BPFILTER_FUNCTION_MAXNAMELEN,
+};
+
+int bpfilter_ipv4_register_targets(void)
+{
+ int err = bpfilter_target_add(&std_tgt);
+
+ if (err)
+ return err;
+ return bpfilter_target_add(&err_tgt);
+}
+
diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile
index 47a0a66..ed5f53b 100644
--- a/net/ipv4/Makefile
+++ b/net/ipv4/Makefile
@@ -15,6 +15,8 @@ obj-y := route.o inetpeer.o protocol.o \
fib_frontend.o fib_semantics.o fib_trie.o fib_notifier.o \
inet_fragment.o ping.o ip_tunnel_core.o gre_offload.o
+obj-$(CONFIG_BPFILTER) += bpfilter/
+
obj-$(CONFIG_NET_IP_TUNNEL) += ip_tunnel.o
obj-$(CONFIG_SYSCTL) += sysctl_net_ipv4.o
obj-$(CONFIG_PROC_FS) += proc.o
diff --git a/net/ipv4/bpfilter/Makefile b/net/ipv4/bpfilter/Makefile
new file mode 100644
index 0000000..ce262d7
--- /dev/null
+++ b/net/ipv4/bpfilter/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_BPFILTER) += sockopt.o
+
diff --git a/net/ipv4/bpfilter/sockopt.c b/net/ipv4/bpfilter/sockopt.c
new file mode 100644
index 0000000..26e544f
--- /dev/null
+++ b/net/ipv4/bpfilter/sockopt.c
@@ -0,0 +1,49 @@
+#include <linux/uaccess.h>
+#include <linux/bpfilter.h>
+#include <uapi/linux/bpf.h>
+#include <linux/wait.h>
+#include <linux/kmod.h>
+struct sock;
+
+extern struct wait_queue_head bpfilter_get_cmd_wq;
+extern struct wait_queue_head bpfilter_reply_wq;
+extern bool bpfilter_get_cmd_ready;
+extern bool bpfilter_reply_ready;
+extern struct bpfilter_get_cmd bpfilter_get_cmd_mbox;
+extern struct bpfilter_reply bpfilter_reply_mbox;
+
+bool loaded = false;
+
+int bpfilter_ip_set_sockopt(struct sock *sk, int optname, char __user *optval,
+ unsigned int optlen)
+{
+ int err;
+
+ if (!loaded) {
+ err = request_module("bpfilter");
+ printk("request_module %d\n", err);
+// if (err)
+// return err;
+ loaded = true;
+ }
+ bpfilter_get_cmd_mbox.pid = current->pid;
+ bpfilter_get_cmd_mbox.cmd = optname;
+ bpfilter_get_cmd_mbox.addr = (long) optval;
+ bpfilter_get_cmd_mbox.len = optlen;
+ bpfilter_get_cmd_ready = true;
+ wake_up(&bpfilter_get_cmd_wq);
+ wait_event_killable(bpfilter_reply_wq, bpfilter_reply_ready);
+ bpfilter_reply_ready = false;
+ return bpfilter_reply_mbox.status;
+}
+
+int bpfilter_ip_get_sockopt(struct sock *sk, int optname, char __user *optval,
+ int __user *optlen)
+{
+ int len;
+
+ if (get_user(len, optlen))
+ return -EFAULT;
+
+ return bpfilter_ip_set_sockopt(sk, optname, optval, len);
+}
diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c
index 6cc70fa..439c1b9 100644
--- a/net/ipv4/ip_sockglue.c
+++ b/net/ipv4/ip_sockglue.c
@@ -47,6 +47,8 @@
#include <linux/errqueue.h>
#include <linux/uaccess.h>
+#include <linux/bpfilter.h>
+
/*
* SOL_IP control messages.
*/
@@ -1250,6 +1252,11 @@ int ip_setsockopt(struct sock *sk, int level,
return -ENOPROTOOPT;
err = do_ip_setsockopt(sk, level, optname, optval, optlen);
+#ifdef CONFIG_BPFILTER
+ if (optname >= BPFILTER_IPT_SO_SET_REPLACE &&
+ optname < BPFILTER_IPT_SET_MAX)
+ err = bpfilter_ip_set_sockopt(sk, optname, optval, optlen);
+#endif
#ifdef CONFIG_NETFILTER
/* we need to exclude all possible ENOPROTOOPTs except default case */
if (err == -ENOPROTOOPT && optname != IP_HDRINCL &&
@@ -1564,6 +1571,11 @@ int ip_getsockopt(struct sock *sk, int level,
int err;
err = do_ip_getsockopt(sk, level, optname, optval, optlen, 0);
+#ifdef CONFIG_BPFILTER
+ if (optname >= BPFILTER_IPT_SO_GET_INFO &&
+ optname < BPFILTER_IPT_GET_MAX)
+ err = bpfilter_ip_get_sockopt(sk, optname, optval, optlen);
+#endif
#ifdef CONFIG_NETFILTER
/* we need to exclude all possible ENOPROTOOPTs except default case */
if (err == -ENOPROTOOPT && optname != IP_PKTOPTIONS &&
@@ -1599,6 +1611,11 @@ int compat_ip_getsockopt(struct sock *sk, int level, int optname,
err = do_ip_getsockopt(sk, level, optname, optval, optlen,
MSG_CMSG_COMPAT);
+#ifdef CONFIG_BPFILTER
+ if (optname >= BPFILTER_IPT_SO_GET_INFO &&
+ optname < BPFILTER_IPT_GET_MAX)
+ err = bpfilter_ip_get_sockopt(sk, optname, optval, optlen);
+#endif
#ifdef CONFIG_NETFILTER
/* we need to exclude all possible ENOPROTOOPTs except default case */
if (err == -ENOPROTOOPT && optname != IP_PKTOPTIONS &&
--
2.9.5
^ permalink raw reply related [flat|nested] 73+ messages in thread
* [PATCH RFC 4/4] bpf: rough bpfilter codegen example hack
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
` (2 preceding siblings ...)
2018-02-16 13:40 ` [PATCH RFC 3/4] net: initial bpfilter skeleton Daniel Borkmann
@ 2018-02-16 13:40 ` Daniel Borkmann
2018-02-16 14:57 ` [PATCH RFC 0/4] net: add bpfilter Florian Westphal
` (2 subsequent siblings)
6 siblings, 0 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-16 13:40 UTC (permalink / raw)
To: netdev; +Cc: netfilter-devel, davem, alexei.starovoitov, Daniel Borkmann
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
include/uapi/linux/bpf.h | 31 +++--
kernel/bpf/syscall.c | 39 +++---
net/bpfilter/Makefile | 2 +-
net/bpfilter/bpfilter.c | 59 +++++----
net/bpfilter/bpfilter_mod.h | 285 ++++++++++++++++++++++++++++++++++++++++++-
net/bpfilter/ctor.c | 57 +++++----
net/bpfilter/gen.c | 290 ++++++++++++++++++++++++++++++++++++++++++++
net/bpfilter/init.c | 11 +-
net/bpfilter/sockopt.c | 137 ++++++++++++++++-----
net/bpfilter/tables.c | 5 +-
net/bpfilter/tgts.c | 1 +
net/ipv4/bpfilter/sockopt.c | 25 +++-
13 files changed, 835 insertions(+), 109 deletions(-)
create mode 100644 net/bpfilter/gen.c
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index ea977e9..066d76b 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -94,8 +94,8 @@ enum bpf_cmd {
BPF_MAP_GET_FD_BY_ID,
BPF_OBJ_GET_INFO_BY_FD,
BPF_PROG_QUERY,
- BPFILTER_GET_CMD,
- BPFILTER_REPLY,
+ BPF_MBOX_REQUEST,
+ BPF_MBOX_REPLY,
};
enum bpf_map_type {
@@ -233,14 +233,29 @@ enum bpf_attach_type {
#define BPF_F_RDONLY (1U << 3)
#define BPF_F_WRONLY (1U << 4)
-struct bpfilter_get_cmd {
- __u32 pid;
- __u32 cmd;
+enum bpf_mbox_subsys {
+ BPF_MBOX_SUBSYS_BPFILTER,
+#define BPF_MBOX_SUBSYS_BPFILTER BPF_MBOX_SUBSYS_BPFILTER
+};
+
+enum bpf_mbox_kind {
+ BPF_MBOX_KIND_SET,
+#define BPF_MBOX_KIND_SET BPF_MBOX_KIND_SET
+ BPF_MBOX_KIND_GET,
+#define BPF_MBOX_KIND_GET BPF_MBOX_KIND_GET
+};
+
+struct bpf_mbox_request {
__u64 addr;
__u32 len;
+ __u32 subsys;
+ __u32 kind;
+ __u32 cmd;
+ __u32 pid;
};
-struct bpfilter_reply {
+struct bpf_mbox_reply {
+ __u32 subsys;
__u32 status;
};
@@ -334,8 +349,8 @@ union bpf_attr {
__u32 prog_cnt;
} query;
- struct bpfilter_get_cmd bpfilter_get_cmd;
- struct bpfilter_reply bpfilter_reply;
+ struct bpf_mbox_request mbox_request;
+ struct bpf_mbox_reply mbox_reply;
} __attribute__((aligned(8)));
/* BPF helper function descriptions:
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index e933bf9..2feb438 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1842,36 +1842,47 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,
DECLARE_WAIT_QUEUE_HEAD(bpfilter_get_cmd_wq);
DECLARE_WAIT_QUEUE_HEAD(bpfilter_reply_wq);
+
bool bpfilter_get_cmd_ready = false;
bool bpfilter_reply_ready = false;
-struct bpfilter_get_cmd bpfilter_get_cmd_mbox;
-struct bpfilter_reply bpfilter_reply_mbox;
-#define BPFILTER_GET_CMD_LAST_FIELD bpfilter_get_cmd.len
+struct bpf_mbox_request bpfilter_get_cmd_mbox;
+struct bpf_mbox_reply bpfilter_reply_mbox;
+
+#define BPF_MBOX_REQUEST_LAST_FIELD mbox_request.pid
-static int bpfilter_get_cmd(const union bpf_attr *attr,
+static int bpf_mbox_request(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
- if (CHECK_ATTR(BPFILTER_GET_CMD))
+ if (CHECK_ATTR(BPF_MBOX_REQUEST))
return -EINVAL;
+ if (attr->mbox_request.subsys != BPF_MBOX_SUBSYS_BPFILTER)
+ return -ENOTSUPP;
+
wait_event_killable(bpfilter_get_cmd_wq, bpfilter_get_cmd_ready);
bpfilter_get_cmd_ready = false;
- if (copy_to_user(&uattr->bpfilter_get_cmd, &bpfilter_get_cmd_mbox,
+
+ if (copy_to_user(&uattr->mbox_request, &bpfilter_get_cmd_mbox,
sizeof(bpfilter_get_cmd_mbox)))
return -EFAULT;
return 0;
}
-#define BPFILTER_REPLY_LAST_FIELD bpfilter_reply.status
+#define BPF_MBOX_REPLY_LAST_FIELD mbox_reply.status
-static int bpfilter_reply(const union bpf_attr *attr,
+static int bpf_mbox_reply(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
- if (CHECK_ATTR(BPFILTER_REPLY))
+ if (CHECK_ATTR(BPF_MBOX_REPLY))
return -EINVAL;
- bpfilter_reply_mbox.status = attr->bpfilter_reply.status;
+ if (attr->mbox_reply.subsys != BPF_MBOX_SUBSYS_BPFILTER)
+ return -ENOTSUPP;
+
+ bpfilter_reply_mbox.subsys = attr->mbox_reply.subsys;
+ bpfilter_reply_mbox.status = attr->mbox_reply.status;
bpfilter_reply_ready = true;
wake_up(&bpfilter_reply_wq);
+
return 0;
}
@@ -1952,11 +1963,11 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
case BPF_OBJ_GET_INFO_BY_FD:
err = bpf_obj_get_info_by_fd(&attr, uattr);
break;
- case BPFILTER_GET_CMD:
- err = bpfilter_get_cmd(&attr, uattr);
+ case BPF_MBOX_REQUEST:
+ err = bpf_mbox_request(&attr, uattr);
break;
- case BPFILTER_REPLY:
- err = bpfilter_reply(&attr, uattr);
+ case BPF_MBOX_REPLY:
+ err = bpf_mbox_reply(&attr, uattr);
break;
default:
err = -EINVAL;
diff --git a/net/bpfilter/Makefile b/net/bpfilter/Makefile
index 5e05505..5a85ef7 100644
--- a/net/bpfilter/Makefile
+++ b/net/bpfilter/Makefile
@@ -5,5 +5,5 @@
hostprogs-y := bpfilter.ko
always := $(hostprogs-y)
-bpfilter.ko-objs := bpfilter.o tgts.o targets.o tables.o init.o ctor.o sockopt.o
+bpfilter.ko-objs := bpfilter.o tgts.o targets.o tables.o init.o ctor.o sockopt.o gen.o
HOSTCFLAGS += -I. -Itools/include/
diff --git a/net/bpfilter/bpfilter.c b/net/bpfilter/bpfilter.c
index 445ae65..364c66a 100644
--- a/net/bpfilter/bpfilter.c
+++ b/net/bpfilter/bpfilter.c
@@ -1,19 +1,22 @@
// SPDX-License-Identifier: GPL-2.0
#define _GNU_SOURCE
-#include <sys/uio.h>
#include <errno.h>
#include <stdio.h>
-#include <sys/socket.h>
#include <fcntl.h>
#include <unistd.h>
-#include "include/uapi/linux/bpf.h"
+
+#include <sys/uio.h>
+#include <sys/socket.h>
+
#include <asm/unistd.h>
+
+#include "include/uapi/linux/bpf.h"
+
#include "bpfilter_mod.h"
extern long int syscall (long int __sysno, ...);
-static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr,
- unsigned int size)
+int sys_bpf(int cmd, union bpf_attr *attr, unsigned int size)
{
return syscall(321, cmd, attr, size);
}
@@ -38,21 +41,35 @@ int copy_to_user(void *addr, const void *src, int len)
struct iovec local;
struct iovec remote;
- local.iov_base = (void *) src;
+ local.iov_base = (void *)src;
local.iov_len = len;
remote.iov_base = addr;
remote.iov_len = len;
return process_vm_writev(pid, &local, 1, &remote, 1, 0) != len;
}
-static int handle_cmd(struct bpfilter_get_cmd *cmd)
+static int handle_get_cmd(struct bpf_mbox_request *cmd)
{
pid = cmd->pid;
switch (cmd->cmd) {
case BPFILTER_IPT_SO_GET_INFO:
- return bpfilter_get_info((void *) (long) cmd->addr, cmd->len);
+ return bpfilter_get_info((void *)(long)cmd->addr, cmd->len);
case BPFILTER_IPT_SO_GET_ENTRIES:
- return bpfilter_get_entries((void *) (long) cmd->addr, cmd->len);
+ return bpfilter_get_entries((void *)(long)cmd->addr, cmd->len);
+ default:
+ break;
+ }
+ return -ENOPROTOOPT;
+}
+
+static int handle_set_cmd(struct bpf_mbox_request *cmd)
+{
+ pid = cmd->pid;
+ switch (cmd->cmd) {
+ case BPFILTER_IPT_SO_SET_REPLACE:
+ return bpfilter_set_replace((void *)(long)cmd->addr, cmd->len);
+ case BPFILTER_IPT_SO_SET_ADD_COUNTERS:
+ return bpfilter_set_add_counters((void *)(long)cmd->addr, cmd->len);
default:
break;
}
@@ -65,24 +82,24 @@ static void loop(void)
bpfilter_ipv4_init();
while (1) {
- union bpf_attr get_cmd = {};
- union bpf_attr reply = {};
- struct bpfilter_get_cmd *cmd;
-
- sys_bpf(BPFILTER_GET_CMD, &get_cmd, sizeof(get_cmd));
- cmd = &get_cmd.bpfilter_get_cmd;
-
- dprintf(debug_fd, "pid %d cmd %d addr %llx len %d\n",
- cmd->pid, cmd->cmd, cmd->addr, cmd->len);
+ union bpf_attr req = {};
+ union bpf_attr rep = {};
+ struct bpf_mbox_request *cmd;
- reply.bpfilter_reply.status = handle_cmd(cmd);
- sys_bpf(BPFILTER_REPLY, &reply, sizeof(reply));
+ req.mbox_request.subsys = BPF_MBOX_SUBSYS_BPFILTER;
+ sys_bpf(BPF_MBOX_REQUEST, &req, sizeof(req));
+ cmd = &req.mbox_request;
+ rep.mbox_reply.subsys = BPF_MBOX_SUBSYS_BPFILTER;
+ rep.mbox_reply.status = cmd->kind == BPF_MBOX_KIND_SET ?
+ handle_set_cmd(cmd) :
+ handle_get_cmd(cmd);
+ sys_bpf(BPF_MBOX_REPLY, &rep, sizeof(rep));
}
}
int main(void)
{
- debug_fd = open("/tmp/aa", 00000002 | 00000100);
+ debug_fd = open("/dev/pts/1" /* /tmp/aa */, 00000002 | 00000100);
loop();
close(debug_fd);
return 0;
diff --git a/net/bpfilter/bpfilter_mod.h b/net/bpfilter/bpfilter_mod.h
index f0de41b..b420998 100644
--- a/net/bpfilter/bpfilter_mod.h
+++ b/net/bpfilter/bpfilter_mod.h
@@ -21,8 +21,8 @@ struct bpfilter_table_info {
unsigned int initial_entries;
unsigned int hook_entry[BPFILTER_INET_HOOK_MAX];
unsigned int underflow[BPFILTER_INET_HOOK_MAX];
- unsigned int stacksize;
- void ***jumpstack;
+// unsigned int stacksize;
+// void ***jumpstack;
unsigned char entries[0] __aligned(8);
};
@@ -64,22 +64,55 @@ struct bpfilter_ipt_error {
struct bpfilter_target {
struct list_head all_target_list;
- const char name[BPFILTER_EXTENSION_MAXNAMELEN];
+ char name[BPFILTER_EXTENSION_MAXNAMELEN];
unsigned int size;
int hold;
u16 family;
u8 rev;
};
+struct bpfilter_gen_ctx {
+ struct bpf_insn *img;
+ u32 len_cur;
+ u32 len_max;
+ u32 default_verdict;
+ int fd;
+ int ifindex;
+ bool offloaded;
+};
+
+union bpf_attr;
+int sys_bpf(int cmd, union bpf_attr *attr, unsigned int size);
+
+int bpfilter_gen_init(struct bpfilter_gen_ctx *ctx);
+int bpfilter_gen_prologue(struct bpfilter_gen_ctx *ctx);
+int bpfilter_gen_epilogue(struct bpfilter_gen_ctx *ctx);
+int bpfilter_gen_append(struct bpfilter_gen_ctx *ctx,
+ struct bpfilter_ipt_ip *ent, int verdict);
+int bpfilter_gen_commit(struct bpfilter_gen_ctx *ctx);
+void bpfilter_gen_destroy(struct bpfilter_gen_ctx *ctx);
+
struct bpfilter_target *bpfilter_target_get_by_name(const char *name);
void bpfilter_target_put(struct bpfilter_target *tgt);
int bpfilter_target_add(struct bpfilter_target *tgt);
-struct bpfilter_table_info *bpfilter_ipv4_table_ctor(struct bpfilter_table *tbl);
+struct bpfilter_table_info *
+bpfilter_ipv4_table_alloc(struct bpfilter_table *tbl, __u32 size_ents);
+struct bpfilter_table_info *
+bpfilter_ipv4_table_finalize(struct bpfilter_table *tbl,
+ struct bpfilter_table_info *info,
+ __u32 size_ents, __u32 num_ents);
+struct bpfilter_table_info *
+bpfilter_ipv4_table_finalize2(struct bpfilter_table *tbl,
+ struct bpfilter_table_info *info,
+ __u32 size_ents, __u32 num_ents);
+
int bpfilter_ipv4_register_targets(void);
void bpfilter_tables_init(void);
int bpfilter_get_info(void *addr, int len);
int bpfilter_get_entries(void *cmd, int len);
+int bpfilter_set_replace(void *cmd, int len);
+int bpfilter_set_add_counters(void *cmd, int len);
int bpfilter_ipv4_init(void);
int copy_from_user(void *dst, void *addr, int len);
@@ -93,4 +126,248 @@ extern int pid;
extern int debug_fd;
#define ENOTSUPP 524
+/* Helper macros for filter block array initializers. */
+
+/* ALU ops on registers, bpf_add|sub|...: dst_reg += src_reg */
+
+#define BPF_ALU64_REG(OP, DST, SRC) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_OP(OP) | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = 0 })
+
+#define BPF_ALU32_REG(OP, DST, SRC) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_OP(OP) | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = 0 })
+
+/* ALU ops on immediates, bpf_add|sub|...: dst_reg += imm32 */
+
+#define BPF_ALU64_IMM(OP, DST, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_OP(OP) | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+#define BPF_ALU32_IMM(OP, DST, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_OP(OP) | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+/* Endianess conversion, cpu_to_{l,b}e(), {l,b}e_to_cpu() */
+
+#define BPF_ENDIAN(TYPE, DST, LEN) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_END | BPF_SRC(TYPE), \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = LEN })
+
+/* Short form of mov, dst_reg = src_reg */
+
+#define BPF_MOV64_REG(DST, SRC) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_MOV | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = 0 })
+
+#define BPF_MOV32_REG(DST, SRC) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_MOV | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = 0 })
+
+/* Short form of mov, dst_reg = imm32 */
+
+#define BPF_MOV64_IMM(DST, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_MOV | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+#define BPF_MOV32_IMM(DST, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_MOV | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+/* BPF_LD_IMM64 macro encodes single 'load 64-bit immediate' insn */
+#define BPF_LD_IMM64(DST, IMM) \
+ BPF_LD_IMM64_RAW(DST, 0, IMM)
+
+#define BPF_LD_IMM64_RAW(DST, SRC, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_LD | BPF_DW | BPF_IMM, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = (__u32) (IMM) }), \
+ ((struct bpf_insn) { \
+ .code = 0, /* zero is reserved opcode */ \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = ((__u64) (IMM)) >> 32 })
+
+/* pseudo BPF_LD_IMM64 insn used to refer to process-local map_fd */
+#define BPF_LD_MAP_FD(DST, MAP_FD) \
+ BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
+
+/* Short form of mov based on type, BPF_X: dst_reg = src_reg, BPF_K: dst_reg = imm32 */
+
+#define BPF_MOV64_RAW(TYPE, DST, SRC, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU64 | BPF_MOV | BPF_SRC(TYPE), \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = IMM })
+
+#define BPF_MOV32_RAW(TYPE, DST, SRC, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ALU | BPF_MOV | BPF_SRC(TYPE), \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = IMM })
+
+/* Direct packet access, R0 = *(uint *) (skb->data + imm32) */
+
+#define BPF_LD_ABS(SIZE, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_LD | BPF_SIZE(SIZE) | BPF_ABS, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = IMM })
+
+/* Indirect packet access, R0 = *(uint *) (skb->data + src_reg + imm32) */
+
+#define BPF_LD_IND(SIZE, SRC, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_LD | BPF_SIZE(SIZE) | BPF_IND, \
+ .dst_reg = 0, \
+ .src_reg = SRC, \
+ .off = 0, \
+ .imm = IMM })
+
+/* Memory load, dst_reg = *(uint *) (src_reg + off16) */
+
+#define BPF_LDX_MEM(SIZE, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_LDX | BPF_SIZE(SIZE) | BPF_MEM, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Memory store, *(uint *) (dst_reg + off16) = src_reg */
+
+#define BPF_STX_MEM(SIZE, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_STX | BPF_SIZE(SIZE) | BPF_MEM, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Atomic memory add, *(uint *)(dst_reg + off16) += src_reg */
+
+#define BPF_STX_XADD(SIZE, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_STX | BPF_SIZE(SIZE) | BPF_XADD, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Memory store, *(uint *) (dst_reg + off16) = imm32 */
+
+#define BPF_ST_MEM(SIZE, DST, OFF, IMM) \
+ ((struct bpf_insn) { \
+ .code = BPF_ST | BPF_SIZE(SIZE) | BPF_MEM, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = OFF, \
+ .imm = IMM })
+
+/* Conditional jumps against registers, if (dst_reg 'op' src_reg) goto pc + off16 */
+
+#define BPF_JMP_REG(OP, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_OP(OP) | BPF_X, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Conditional jumps against immediates, if (dst_reg 'op' imm32) goto pc + off16 */
+
+#define BPF_JMP_IMM(OP, DST, IMM, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_OP(OP) | BPF_K, \
+ .dst_reg = DST, \
+ .src_reg = 0, \
+ .off = OFF, \
+ .imm = IMM })
+
+/* Unconditional jumps, goto pc + off16 */
+
+#define BPF_JMP_A(OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_JA, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = OFF, \
+ .imm = 0 })
+
+/* Function call */
+
+#define BPF_EMIT_CALL(FUNC) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_CALL, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = ((FUNC) - __bpf_call_base) })
+
+/* Raw code statement block */
+
+#define BPF_RAW_INSN(CODE, DST, SRC, OFF, IMM) \
+ ((struct bpf_insn) { \
+ .code = CODE, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = IMM })
+
+/* Program exit */
+
+#define BPF_EXIT_INSN() \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_EXIT, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = 0, \
+ .imm = 0 })
+
#endif
diff --git a/net/bpfilter/ctor.c b/net/bpfilter/ctor.c
index efb7fee..ba44c21 100644
--- a/net/bpfilter/ctor.c
+++ b/net/bpfilter/ctor.c
@@ -1,8 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
-#include <sys/socket.h>
-#include <linux/bitops.h>
#include <stdlib.h>
#include <stdio.h>
+#include <string.h>
+
+#include <sys/socket.h>
+
+#include <linux/bitops.h>
+
#include "bpfilter_mod.h"
unsigned int __sw_hweight32(unsigned int w)
@@ -13,35 +17,47 @@ unsigned int __sw_hweight32(unsigned int w)
return (w * 0x01010101) >> 24;
}
-struct bpfilter_table_info *bpfilter_ipv4_table_ctor(struct bpfilter_table *tbl)
+struct bpfilter_table_info *bpfilter_ipv4_table_alloc(struct bpfilter_table *tbl,
+ __u32 size_ents)
{
unsigned int num_hooks = hweight32(tbl->valid_hooks);
- struct bpfilter_ipt_standard *tgts;
struct bpfilter_table_info *info;
- struct bpfilter_ipt_error *term;
- unsigned int mask, offset, h, i;
unsigned int size, alloc_size;
size = sizeof(struct bpfilter_ipt_standard) * num_hooks;
size += sizeof(struct bpfilter_ipt_error);
+ size += size_ents;
alloc_size = size + sizeof(struct bpfilter_table_info);
info = malloc(alloc_size);
- if (!info)
- return NULL;
+ if (info) {
+ memset(info, 0, alloc_size);
+ info->size = size;
+ }
+ return info;
+}
+
+struct bpfilter_table_info *bpfilter_ipv4_table_finalize(struct bpfilter_table *tbl,
+ struct bpfilter_table_info *info,
+ __u32 size_ents, __u32 num_ents)
+{
+ unsigned int num_hooks = hweight32(tbl->valid_hooks);
+ struct bpfilter_ipt_standard *tgts;
+ struct bpfilter_ipt_error *term;
+ struct bpfilter_ipt_entry *ent;
+ unsigned int mask, offset, h, i;
- info->num_entries = num_hooks + 1;
- info->size = size;
+ info->num_entries = num_ents + num_hooks + 1;
- tgts = (struct bpfilter_ipt_standard *) (info + 1);
- term = (struct bpfilter_ipt_error *) (tgts + num_hooks);
+ ent = (struct bpfilter_ipt_entry *)(info + 1);
+ tgts = (struct bpfilter_ipt_standard *)((u8 *)ent + size_ents);
+ term = (struct bpfilter_ipt_error *)(tgts + num_hooks);
mask = tbl->valid_hooks;
offset = 0;
h = 0;
i = 0;
- dprintf(debug_fd, "mask %x num_hooks %d\n", mask, num_hooks);
while (mask) {
struct bpfilter_ipt_standard *t;
@@ -55,7 +71,6 @@ struct bpfilter_table_info *bpfilter_ipv4_table_ctor(struct bpfilter_table *tbl)
BPFILTER_IPT_STANDARD_INIT(BPFILTER_NF_ACCEPT);
t->target.target.u.kernel.target =
bpfilter_target_get_by_name(t->target.target.u.user.name);
- dprintf(debug_fd, "user.name %s\n", t->target.target.u.user.name);
if (!t->target.target.u.kernel.target)
goto out_fail;
@@ -67,14 +82,10 @@ struct bpfilter_table_info *bpfilter_ipv4_table_ctor(struct bpfilter_table *tbl)
*term = (struct bpfilter_ipt_error) BPFILTER_IPT_ERROR_INIT;
term->target.target.u.kernel.target =
bpfilter_target_get_by_name(term->target.target.u.user.name);
- dprintf(debug_fd, "user.name %s\n", term->target.target.u.user.name);
- if (!term->target.target.u.kernel.target)
- goto out_fail;
-
- dprintf(debug_fd, "info %p\n", info);
- return info;
-
+ if (!term->target.target.u.kernel.target) {
out_fail:
- free(info);
- return NULL;
+ free(info);
+ return NULL;
+ }
+ return info;
}
diff --git a/net/bpfilter/gen.c b/net/bpfilter/gen.c
new file mode 100644
index 0000000..8e08561
--- /dev/null
+++ b/net/bpfilter/gen.c
@@ -0,0 +1,290 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <errno.h>
+#include <string.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <linux/if_ether.h>
+#include <linux/if_link.h>
+#include <linux/rtnetlink.h>
+#include <linux/bpf.h>
+typedef __u16 __bitwise __sum16; /* hack */
+#include <linux/ip.h>
+
+#include <arpa/inet.h>
+
+#include "bpfilter_mod.h"
+
+unsigned int if_nametoindex(const char *ifname);
+
+static inline __u64 bpf_ptr_to_u64(const void *ptr)
+{
+ return (__u64)(unsigned long)ptr;
+}
+
+static int bpf_prog_load(enum bpf_prog_type type,
+ const struct bpf_insn *insns,
+ unsigned int insn_num,
+ __u32 offload_ifindex)
+{
+ union bpf_attr attr = {};
+
+ attr.prog_type = type;
+ attr.insns = bpf_ptr_to_u64(insns);
+ attr.insn_cnt = insn_num;
+ attr.license = bpf_ptr_to_u64("GPL");
+ attr.prog_ifindex = offload_ifindex;
+
+ return sys_bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
+}
+
+static int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags)
+{
+ struct sockaddr_nl sa;
+ int sock, seq = 0, len, ret = -1;
+ char buf[4096];
+ struct nlattr *nla, *nla_xdp;
+ struct {
+ struct nlmsghdr nh;
+ struct ifinfomsg ifinfo;
+ char attrbuf[64];
+ } req;
+ struct nlmsghdr *nh;
+ struct nlmsgerr *err;
+
+ memset(&sa, 0, sizeof(sa));
+ sa.nl_family = AF_NETLINK;
+
+ sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
+ if (sock < 0) {
+ printf("open netlink socket: %s\n", strerror(errno));
+ return -1;
+ }
+
+ if (bind(sock, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
+ printf("bind to netlink: %s\n", strerror(errno));
+ goto cleanup;
+ }
+
+ memset(&req, 0, sizeof(req));
+ req.nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg));
+ req.nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
+ req.nh.nlmsg_type = RTM_SETLINK;
+ req.nh.nlmsg_pid = 0;
+ req.nh.nlmsg_seq = ++seq;
+ req.ifinfo.ifi_family = AF_UNSPEC;
+ req.ifinfo.ifi_index = ifindex;
+
+ /* started nested attribute for XDP */
+ nla = (struct nlattr *)(((char *)&req)
+ + NLMSG_ALIGN(req.nh.nlmsg_len));
+ nla->nla_type = NLA_F_NESTED | 43/*IFLA_XDP*/;
+ nla->nla_len = NLA_HDRLEN;
+
+ /* add XDP fd */
+ nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
+ nla_xdp->nla_type = 1/*IFLA_XDP_FD*/;
+ nla_xdp->nla_len = NLA_HDRLEN + sizeof(int);
+ memcpy((char *)nla_xdp + NLA_HDRLEN, &fd, sizeof(fd));
+ nla->nla_len += nla_xdp->nla_len;
+
+ /* if user passed in any flags, add those too */
+ if (flags) {
+ nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
+ nla_xdp->nla_type = 3/*IFLA_XDP_FLAGS*/;
+ nla_xdp->nla_len = NLA_HDRLEN + sizeof(flags);
+ memcpy((char *)nla_xdp + NLA_HDRLEN, &flags, sizeof(flags));
+ nla->nla_len += nla_xdp->nla_len;
+ }
+
+ req.nh.nlmsg_len += NLA_ALIGN(nla->nla_len);
+
+ if (send(sock, &req, req.nh.nlmsg_len, 0) < 0) {
+ printf("send to netlink: %s\n", strerror(errno));
+ goto cleanup;
+ }
+
+ len = recv(sock, buf, sizeof(buf), 0);
+ if (len < 0) {
+ printf("recv from netlink: %s\n", strerror(errno));
+ goto cleanup;
+ }
+
+ for (nh = (struct nlmsghdr *)buf; NLMSG_OK(nh, len);
+ nh = NLMSG_NEXT(nh, len)) {
+ if (nh->nlmsg_pid != getpid()) {
+ printf("Wrong pid %d, expected %d\n",
+ nh->nlmsg_pid, getpid());
+ goto cleanup;
+ }
+ if (nh->nlmsg_seq != seq) {
+ printf("Wrong seq %d, expected %d\n",
+ nh->nlmsg_seq, seq);
+ goto cleanup;
+ }
+ switch (nh->nlmsg_type) {
+ case NLMSG_ERROR:
+ err = (struct nlmsgerr *)NLMSG_DATA(nh);
+ if (!err->error)
+ continue;
+ printf("nlmsg error %s\n", strerror(-err->error));
+ goto cleanup;
+ case NLMSG_DONE:
+ break;
+ }
+ }
+
+ ret = 0;
+
+cleanup:
+ close(sock);
+ return ret;
+}
+
+static int bpfilter_load_dev(struct bpfilter_gen_ctx *ctx)
+{
+ u32 xdp_flags = 0;
+
+ if (ctx->offloaded)
+ xdp_flags |= XDP_FLAGS_HW_MODE;
+ return bpf_set_link_xdp_fd(ctx->ifindex, ctx->fd, xdp_flags);
+}
+
+int bpfilter_gen_init(struct bpfilter_gen_ctx *ctx)
+{
+ unsigned int len_max = BPF_MAXINSNS;
+
+ memset(ctx, 0, sizeof(*ctx));
+ ctx->img = calloc(len_max, sizeof(struct bpf_insn));
+ if (!ctx->img)
+ return -ENOMEM;
+ ctx->len_max = len_max;
+ ctx->fd = -1;
+ ctx->default_verdict = XDP_PASS;
+
+ return 0;
+}
+
+#define EMIT(x) \
+ do { \
+ if (ctx->len_cur + 1 > ctx->len_max) \
+ return -ENOMEM; \
+ ctx->img[ctx->len_cur++] = x; \
+ } while (0)
+
+int bpfilter_gen_prologue(struct bpfilter_gen_ctx *ctx)
+{
+ EMIT(BPF_MOV64_REG(BPF_REG_9, BPF_REG_1));
+ EMIT(BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_9,
+ offsetof(struct xdp_md, data)));
+ EMIT(BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_9,
+ offsetof(struct xdp_md, data_end)));
+ EMIT(BPF_MOV64_REG(BPF_REG_1, BPF_REG_2));
+ EMIT(BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, ETH_HLEN));
+ EMIT(BPF_JMP_REG(BPF_JLE, BPF_REG_1, BPF_REG_3, 2));
+ EMIT(BPF_MOV32_IMM(BPF_REG_0, ctx->default_verdict));
+ EMIT(BPF_EXIT_INSN());
+ return 0;
+}
+
+int bpfilter_gen_epilogue(struct bpfilter_gen_ctx *ctx)
+{
+ EMIT(BPF_MOV32_IMM(BPF_REG_0, ctx->default_verdict));
+ EMIT(BPF_EXIT_INSN());
+ return 0;
+}
+
+static int bpfilter_gen_check_entry(const struct bpfilter_ipt_ip *ent)
+{
+#define M_FF "\xff\xff\xff\xff"
+ static const __u8 mask1[IFNAMSIZ] = M_FF M_FF M_FF M_FF;
+ static const __u8 mask0[IFNAMSIZ] = { };
+ int ones = strlen(ent->in_iface); ones += ones > 0;
+#undef M_FF
+ if (strlen(ent->out_iface) > 0)
+ return -ENOTSUPP;
+ if (memcmp(ent->in_iface_mask, mask1, ones) ||
+ memcmp(&ent->in_iface_mask[ones], mask0, sizeof(mask0) - ones))
+ return -ENOTSUPP;
+ if ((ent->src_mask != 0 && ent->src_mask != 0xffffffff) ||
+ (ent->dst_mask != 0 && ent->dst_mask != 0xffffffff))
+ return -ENOTSUPP;
+
+ return 0;
+}
+
+int bpfilter_gen_append(struct bpfilter_gen_ctx *ctx,
+ struct bpfilter_ipt_ip *ent, int verdict)
+{
+ u32 match_xdp = verdict == -1 ? XDP_DROP : XDP_PASS;
+ int ret, ifindex, match_state = 0;
+
+ /* convention R1: tmp, R2: data, R3: data_end, R9: xdp_buff */
+ ret = bpfilter_gen_check_entry(ent);
+ if (ret < 0)
+ return ret;
+ if (ent->src_mask == 0 && ent->dst_mask == 0)
+ return 0;
+
+ ifindex = if_nametoindex(ent->in_iface);
+ if (!ifindex)
+ return 0;
+ if (ctx->ifindex && ctx->ifindex != ifindex)
+ return -ENOTSUPP;
+
+ ctx->ifindex = ifindex;
+ match_state = !!ent->src_mask + !!ent->dst_mask;
+
+ EMIT(BPF_MOV64_REG(BPF_REG_1, BPF_REG_2));
+ EMIT(BPF_MOV32_IMM(BPF_REG_5, 0));
+ EMIT(BPF_LDX_MEM(BPF_H, BPF_REG_4, BPF_REG_1,
+ offsetof(struct ethhdr, h_proto)));
+ EMIT(BPF_JMP_IMM(BPF_JNE, BPF_REG_4, htons(ETH_P_IP),
+ 3 + match_state * 3));
+ EMIT(BPF_ALU64_IMM(BPF_ADD, BPF_REG_1,
+ sizeof(struct ethhdr) + sizeof(struct iphdr)));
+ EMIT(BPF_JMP_REG(BPF_JGT, BPF_REG_1, BPF_REG_3, 1 + match_state * 3));
+ EMIT(BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, -(int)sizeof(struct iphdr)));
+ if (ent->src_mask) {
+ EMIT(BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_1,
+ offsetof(struct iphdr, saddr)));
+ EMIT(BPF_JMP_IMM(BPF_JNE, BPF_REG_4, ent->src, 1));
+ EMIT(BPF_ALU32_IMM(BPF_ADD, BPF_REG_5, 1));
+ }
+ if (ent->dst_mask) {
+ EMIT(BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_1,
+ offsetof(struct iphdr, daddr)));
+ EMIT(BPF_JMP_IMM(BPF_JNE, BPF_REG_4, ent->dst, 1));
+ EMIT(BPF_ALU32_IMM(BPF_ADD, BPF_REG_5, 1));
+ }
+ EMIT(BPF_JMP_IMM(BPF_JNE, BPF_REG_5, match_state, 2));
+ EMIT(BPF_MOV32_IMM(BPF_REG_0, match_xdp));
+ EMIT(BPF_EXIT_INSN());
+ return 0;
+}
+
+int bpfilter_gen_commit(struct bpfilter_gen_ctx *ctx)
+{
+ int ret;
+
+ ret = bpf_prog_load(BPF_PROG_TYPE_XDP, ctx->img,
+ ctx->len_cur, ctx->ifindex);
+ if (ret > 0)
+ ctx->offloaded = true;
+ if (ret < 0)
+ ret = bpf_prog_load(BPF_PROG_TYPE_XDP, ctx->img,
+ ctx->len_cur, 0);
+ if (ret > 0) {
+ ctx->fd = ret;
+ ret = bpfilter_load_dev(ctx);
+ }
+
+ return ret < 0 ? ret : 0;
+}
+
+void bpfilter_gen_destroy(struct bpfilter_gen_ctx *ctx)
+{
+ free(ctx->img);
+ close(ctx->fd);
+}
diff --git a/net/bpfilter/init.c b/net/bpfilter/init.c
index 699f3f6..14e621a 100644
--- a/net/bpfilter/init.c
+++ b/net/bpfilter/init.c
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
-#include <sys/socket.h>
#include <errno.h>
+
+#include <sys/socket.h>
+
#include "bpfilter_mod.h"
static struct bpfilter_table filter_table_ipv4 = {
@@ -22,12 +24,13 @@ int bpfilter_ipv4_init(void)
if (err)
return err;
- info = bpfilter_ipv4_table_ctor(t);
+ info = bpfilter_ipv4_table_alloc(t, 0);
+ if (!info)
+ return -ENOMEM;
+ info = bpfilter_ipv4_table_finalize(t, info, 0, 0);
if (!info)
return -ENOMEM;
-
t->info = info;
-
return bpfilter_table_add(&filter_table_ipv4);
}
diff --git a/net/bpfilter/sockopt.c b/net/bpfilter/sockopt.c
index 43687da..26ad12a 100644
--- a/net/bpfilter/sockopt.c
+++ b/net/bpfilter/sockopt.c
@@ -1,10 +1,14 @@
// SPDX-License-Identifier: GPL-2.0
-#include <sys/socket.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
+#include <stdlib.h>
+
+#include <sys/socket.h>
+
#include "bpfilter_mod.h"
+/* TODO: Get all of this in here properly done in encoding/decoding layer. */
static int fetch_name(void *addr, int len, char *name, int name_len)
{
if (copy_from_user(name, addr, name_len))
@@ -55,12 +59,17 @@ int bpfilter_get_info(void *addr, int len)
return err;
}
-static int copy_target(struct bpfilter_standard_target *ut,
- struct bpfilter_standard_target *kt)
+static int target_u2k(struct bpfilter_standard_target *kt)
{
- struct bpfilter_target *tgt;
- int sz;
+ kt->target.u.kernel.target =
+ bpfilter_target_get_by_name(kt->target.u.user.name);
+ return kt->target.u.kernel.target ? 0 : -EINVAL;
+}
+static int target_k2u(struct bpfilter_standard_target *ut,
+ struct bpfilter_standard_target *kt)
+{
+ struct bpfilter_target *tgt;
if (put_user(kt->target.u.target_size,
&ut->target.u.target_size))
@@ -69,12 +78,9 @@ static int copy_target(struct bpfilter_standard_target *ut,
tgt = kt->target.u.kernel.target;
if (copy_to_user(ut->target.u.user.name, tgt->name, strlen(tgt->name)))
return -EFAULT;
-
if (put_user(tgt->rev, &ut->target.u.user.revision))
return -EFAULT;
-
- sz = tgt->size;
- if (copy_to_user(ut->target.data, kt->target.data, sz))
+ if (copy_to_user(ut->target.data, kt->target.data, tgt->size))
return -EFAULT;
return 0;
@@ -84,30 +90,25 @@ static int do_get_entries(void *up,
struct bpfilter_table *tbl,
struct bpfilter_table_info *info)
{
- unsigned int total_size = info->size;
const struct bpfilter_ipt_entry *ent;
+ unsigned int total_size = info->size;
+ void *base = info->entries;
unsigned int off;
- void *base;
-
- base = info->entries;
for (off = 0; off < total_size; off += ent->next_offset) {
- struct bpfilter_xt_counters *cntrs;
struct bpfilter_standard_target *tgt;
+ struct bpfilter_xt_counters *cntrs;
ent = base + off;
if (copy_to_user(up + off, ent, sizeof(*ent)))
return -EFAULT;
-
- /* XXX Just clear counters for now. XXX */
+ /* XXX: Just clear counters for now. */
cntrs = up + off + offsetof(struct bpfilter_ipt_entry, cntrs);
if (put_user(0, &cntrs->packet_cnt) ||
put_user(0, &cntrs->byte_cnt))
return -EINVAL;
-
- tgt = (void *) ent + ent->target_offset;
- dprintf(debug_fd, "target.verdict %d\n", tgt->verdict);
- if (copy_target(up + off + ent->target_offset, tgt))
+ tgt = (void *)ent + ent->target_offset;
+ if (target_k2u(up + off + ent->target_offset, tgt))
return -EFAULT;
}
return 0;
@@ -123,31 +124,113 @@ int bpfilter_get_entries(void *cmd, int len)
if (len < sizeof(struct bpfilter_ipt_get_entries))
return -EINVAL;
-
if (copy_from_user(&req, cmd, sizeof(req)))
return -EFAULT;
-
tbl = bpfilter_table_get_by_name(req.name, strlen(req.name));
if (!tbl)
return -ENOENT;
-
info = tbl->info;
if (!info) {
err = -ENOENT;
goto out_put;
}
-
if (info->size != req.size) {
err = -EINVAL;
goto out_put;
}
-
err = do_get_entries(uptr->entries, tbl, info);
- dprintf(debug_fd, "do_get_entries %d req.size %d\n", err, req.size);
-
out_put:
bpfilter_table_put(tbl);
+ return err;
+}
+static int do_set_replace(struct bpfilter_ipt_replace *req, void *base,
+ struct bpfilter_table *tbl)
+{
+ unsigned int total_size = req->size;
+ struct bpfilter_table_info *info;
+ struct bpfilter_ipt_entry *ent;
+ struct bpfilter_gen_ctx ctx;
+ unsigned int off, sents = 0, ents = 0;
+ int ret;
+
+ ret = bpfilter_gen_init(&ctx);
+ if (ret < 0)
+ return ret;
+ ret = bpfilter_gen_prologue(&ctx);
+ if (ret < 0)
+ return ret;
+ info = bpfilter_ipv4_table_alloc(tbl, total_size);
+ if (!info)
+ return -ENOMEM;
+ if (copy_from_user(&info->entries[0], base, req->size)) {
+ free(info);
+ return -EFAULT;
+ }
+ base = &info->entries[0];
+ for (off = 0; off < total_size; off += ent->next_offset) {
+ struct bpfilter_standard_target *tgt;
+ ent = base + off;
+ ents++;
+ sents += ent->next_offset;
+ tgt = (void *) ent + ent->target_offset;
+ target_u2k(tgt);
+ ret = bpfilter_gen_append(&ctx, &ent->ip, tgt->verdict);
+ if (ret < 0)
+ goto err;
+ }
+ info->num_entries = ents;
+ info->size = sents;
+ memcpy(info->hook_entry, req->hook_entry, sizeof(info->hook_entry));
+ memcpy(info->underflow, req->underflow, sizeof(info->hook_entry));
+ ret = bpfilter_gen_epilogue(&ctx);
+ if (ret < 0)
+ goto err;
+ ret = bpfilter_gen_commit(&ctx);
+ if (ret < 0)
+ goto err;
+ free(tbl->info);
+ tbl->info = info;
+ bpfilter_gen_destroy(&ctx);
+ dprintf(debug_fd, "offloaded %u\n", ctx.offloaded);
+ return ret;
+err:
+ free(info);
+ return ret;
+}
+
+int bpfilter_set_replace(void *cmd, int len)
+{
+ struct bpfilter_ipt_replace *uptr = cmd;
+ struct bpfilter_ipt_replace req;
+ struct bpfilter_table_info *info;
+ struct bpfilter_table *tbl;
+ int err;
+
+ if (len < sizeof(req))
+ return -EINVAL;
+ if (copy_from_user(&req, cmd, sizeof(req)))
+ return -EFAULT;
+ if (req.num_counters >= INT_MAX / sizeof(struct bpfilter_xt_counters))
+ return -ENOMEM;
+ if (req.num_counters == 0)
+ return -EINVAL;
+ req.name[sizeof(req.name) - 1] = 0;
+ tbl = bpfilter_table_get_by_name(req.name, strlen(req.name));
+ if (!tbl)
+ return -ENOENT;
+ info = tbl->info;
+ if (!info) {
+ err = -ENOENT;
+ goto out_put;
+ }
+ err = do_set_replace(&req, uptr->entries, tbl);
+out_put:
+ bpfilter_table_put(tbl);
return err;
}
+int bpfilter_set_add_counters(void *cmd, int len)
+{
+ return 0;
+}
diff --git a/net/bpfilter/tables.c b/net/bpfilter/tables.c
index 9a96599..e0dab28 100644
--- a/net/bpfilter/tables.c
+++ b/net/bpfilter/tables.c
@@ -1,8 +1,11 @@
// SPDX-License-Identifier: GPL-2.0
-#include <sys/socket.h>
#include <errno.h>
#include <string.h>
+
+#include <sys/socket.h>
+
#include <linux/hashtable.h>
+
#include "bpfilter_mod.h"
static unsigned int full_name_hash(const void *salt, const char *name, unsigned int len)
diff --git a/net/bpfilter/tgts.c b/net/bpfilter/tgts.c
index eac5e8a..0a00bc28 100644
--- a/net/bpfilter/tgts.c
+++ b/net/bpfilter/tgts.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
#include <sys/socket.h>
+
#include "bpfilter_mod.h"
struct bpfilter_target std_tgt = {
diff --git a/net/ipv4/bpfilter/sockopt.c b/net/ipv4/bpfilter/sockopt.c
index 26e544f..159a64580 100644
--- a/net/ipv4/bpfilter/sockopt.c
+++ b/net/ipv4/bpfilter/sockopt.c
@@ -7,15 +7,17 @@ struct sock;
extern struct wait_queue_head bpfilter_get_cmd_wq;
extern struct wait_queue_head bpfilter_reply_wq;
+
extern bool bpfilter_get_cmd_ready;
extern bool bpfilter_reply_ready;
-extern struct bpfilter_get_cmd bpfilter_get_cmd_mbox;
-extern struct bpfilter_reply bpfilter_reply_mbox;
+
+extern struct bpf_mbox_request bpfilter_get_cmd_mbox;
+extern struct bpf_mbox_reply bpfilter_reply_mbox;
bool loaded = false;
-int bpfilter_ip_set_sockopt(struct sock *sk, int optname, char __user *optval,
- unsigned int optlen)
+int bpfilter_mbox_request(struct sock *sk, int optname, char __user *optval,
+ unsigned int optlen, int kind)
{
int err;
@@ -26,17 +28,29 @@ int bpfilter_ip_set_sockopt(struct sock *sk, int optname, char __user *optval,
// return err;
loaded = true;
}
+
+ bpfilter_get_cmd_mbox.subsys = BPF_MBOX_SUBSYS_BPFILTER;
+ bpfilter_get_cmd_mbox.kind = kind;
bpfilter_get_cmd_mbox.pid = current->pid;
bpfilter_get_cmd_mbox.cmd = optname;
bpfilter_get_cmd_mbox.addr = (long) optval;
bpfilter_get_cmd_mbox.len = optlen;
bpfilter_get_cmd_ready = true;
+
wake_up(&bpfilter_get_cmd_wq);
wait_event_killable(bpfilter_reply_wq, bpfilter_reply_ready);
bpfilter_reply_ready = false;
+
return bpfilter_reply_mbox.status;
}
+int bpfilter_ip_set_sockopt(struct sock *sk, int optname, char __user *optval,
+ unsigned int optlen)
+{
+ return bpfilter_mbox_request(sk, optname, optval, optlen,
+ BPF_MBOX_KIND_SET);
+}
+
int bpfilter_ip_get_sockopt(struct sock *sk, int optname, char __user *optval,
int __user *optlen)
{
@@ -45,5 +59,6 @@ int bpfilter_ip_get_sockopt(struct sock *sk, int optname, char __user *optval,
if (get_user(len, optlen))
return -EFAULT;
- return bpfilter_ip_set_sockopt(sk, optname, optval, len);
+ return bpfilter_mbox_request(sk, optname, optval, len,
+ BPF_MBOX_KIND_GET);
}
--
2.9.5
^ permalink raw reply related [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
` (3 preceding siblings ...)
2018-02-16 13:40 ` [PATCH RFC 4/4] bpf: rough bpfilter codegen example hack Daniel Borkmann
@ 2018-02-16 14:57 ` Florian Westphal
2018-02-16 16:14 ` Florian Westphal
` (2 more replies)
2018-02-17 12:11 ` Harald Welte
2018-02-18 23:35 ` Florian Westphal
6 siblings, 3 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-16 14:57 UTC (permalink / raw)
To: Daniel Borkmann; +Cc: netdev, netfilter-devel, davem, alexei.starovoitov
Daniel Borkmann <daniel@iogearbox.net> wrote:
> This is a very rough and early proof of concept that implements bpfilter.
[..]
> Also, as a benefit from such design, we get BPF JIT compilation on x86_64,
> arm64, ppc64, sparc64, mips64, s390x and arm32, but also rule offloading
> into HW for free for Netronome NFP SmartNICs that are already capable of
> offloading BPF since we can reuse all existing BPF infrastructure as the
> back end. The user space iptables binary issuing rule addition or dumps was
> left as-is, thus at some point any binaries against iptables uapi kernel
> interface could transparently be supported in such manner in long term.
>
> As rule translation can potentially become very complex, this is performed
> entirely in user space. In order to ease deployment, request_module() code
> is extended to allow user mode helpers to be invoked. Idea is that user mode
> helpers are built as part of the kernel build and installed as traditional
> kernel modules with .ko file extension into distro specified location,
> such that from a distribution point of view, they are no different than
> regular kernel modules. Thus, allow request_module() logic to load such
> user mode helper (umh) binaries via:
>
> request_module("foo") ->
> call_umh("modprobe foo") ->
> sys_finit_module(FD of /lib/modules/.../foo.ko) ->
> call_umh(struct file)
>
> Such approach enables kernel to delegate functionality traditionally done
> by kernel modules into user space processes (either root or !root) and
> reduces security attack surface of such new code, meaning in case of
> potential bugs only the umh would crash but not the kernel. Another
> advantage coming with that would be that bpfilter.ko can be debugged and
> tested out of user space as well (e.g. opening the possibility to run
> all clang sanitizers, fuzzers or test suites for checking translation).
Several questions spinning at the moment, I will probably come up with
more:
1. Does this still attach the binary blob to the 'normal' iptables
hooks?
2. If yes, do you see issues wrt. 'iptables' and 'bpfilter' attached
programs being different in nature (e.g. changed by different entities)?
3. What happens if the rule can't be translated (yet?)
4. Do you plan to reimplement connection tracking in userspace?
If no, how will the bpf program interact with it?
[ same question applies to ipv6 exthdr traversal, ip defragmentation
and the like ].
I will probably have a quadrillion of followup questions, sorry :-/
> Also, such architecture makes the kernel/user boundary very precise,
> meaning requests can be handled and BPF translated in control plane part
> in user space with its own user memory etc, while minimal data plane
> bits are in kernel. It would also allow to remove old xtables modules
> at some point from the kernel while keeping functionality in place.
This is what we tried with nftables :-/
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 14:57 ` [PATCH RFC 0/4] net: add bpfilter Florian Westphal
@ 2018-02-16 16:14 ` Florian Westphal
2018-02-16 20:44 ` Daniel Borkmann
2018-02-16 22:33 ` David Miller
2018-02-16 16:53 ` Daniel Borkmann
2018-02-16 22:32 ` David Miller
2 siblings, 2 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-16 16:14 UTC (permalink / raw)
To: Florian Westphal
Cc: Daniel Borkmann, netdev, netfilter-devel, davem, alexei.starovoitov
Florian Westphal <fw@strlen.de> wrote:
> Daniel Borkmann <daniel@iogearbox.net> wrote:
> Several questions spinning at the moment, I will probably come up with
> more:
... and here there are some more ...
One of the many pain points of xtables design is the assumption of 'used
only by sysadmin'.
This has not been true for a very long time, so by now iptables has
this userspace lock (yes, its fugly workaround) to serialize concurrent
iptables invocations in userspace.
AFAIU the translate-in-userspace design now brings back the old problem
of different tools overwriting each others iptables rules.
Another question -- am i correct in that each rule manipulation would
incur a 'recompilation'? Or are there different mini programs chained
together?
One of the nftables advantages is that (since rule representation in
kernel is black-box from userspace point of view) is that the kernel
can announce add/delete of rules or elements from nftables sets.
Any particular reason why translating iptables rather than nftables
(it should be possible to monitor the nftables changes that are
announced by kernel and act on those)?
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 14:57 ` [PATCH RFC 0/4] net: add bpfilter Florian Westphal
2018-02-16 16:14 ` Florian Westphal
@ 2018-02-16 16:53 ` Daniel Borkmann
2018-02-16 22:32 ` David Miller
2 siblings, 0 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-16 16:53 UTC (permalink / raw)
To: Florian Westphal; +Cc: netdev, netfilter-devel, davem, alexei.starovoitov
Hi Florian,
thanks for your feedback! More inline:
On 02/16/2018 03:57 PM, Florian Westphal wrote:
> Daniel Borkmann <daniel@iogearbox.net> wrote:
>> This is a very rough and early proof of concept that implements bpfilter.
>
> [..]
>
>> Also, as a benefit from such design, we get BPF JIT compilation on x86_64,
>> arm64, ppc64, sparc64, mips64, s390x and arm32, but also rule offloading
>> into HW for free for Netronome NFP SmartNICs that are already capable of
>> offloading BPF since we can reuse all existing BPF infrastructure as the
>> back end. The user space iptables binary issuing rule addition or dumps was
>> left as-is, thus at some point any binaries against iptables uapi kernel
>> interface could transparently be supported in such manner in long term.
>>
>> As rule translation can potentially become very complex, this is performed
>> entirely in user space. In order to ease deployment, request_module() code
>> is extended to allow user mode helpers to be invoked. Idea is that user mode
>> helpers are built as part of the kernel build and installed as traditional
>> kernel modules with .ko file extension into distro specified location,
>> such that from a distribution point of view, they are no different than
>> regular kernel modules. Thus, allow request_module() logic to load such
>> user mode helper (umh) binaries via:
>>
>> request_module("foo") ->
>> call_umh("modprobe foo") ->
>> sys_finit_module(FD of /lib/modules/.../foo.ko) ->
>> call_umh(struct file)
>>
>> Such approach enables kernel to delegate functionality traditionally done
>> by kernel modules into user space processes (either root or !root) and
>> reduces security attack surface of such new code, meaning in case of
>> potential bugs only the umh would crash but not the kernel. Another
>> advantage coming with that would be that bpfilter.ko can be debugged and
>> tested out of user space as well (e.g. opening the possibility to run
>> all clang sanitizers, fuzzers or test suites for checking translation).
>
> Several questions spinning at the moment, I will probably come up with
> more:
Sure, no problem at all. It's an early RFC, so purpose is to get a
discussion going on such potential approach.
> 1. Does this still attach the binary blob to the 'normal' iptables
> hooks?
Yeah, so thought would be to keep the user land tooling functional as
is w/o having to recompile binaries, thus this would also need to attach
for the existing hooks in order to keep semantics working. As a benefit
in addition we can also reuse all the rest of the infrastructure to utilize
things like XDP for iptables in the background, there is definitely
flexibility on this side thus users could eventually benefit from this
transparently and don't need to know that 'bpfilter' exists and is
translating in the background. I realize taking this path is a long term
undertake that we would need to tackle as a community, not just one or
two individuals when we decide to go for this direction.
> 2. If yes, do you see issues wrt. 'iptables' and 'bpfilter' attached
> programs being different in nature (e.g. changed by different entities)?
There could certainly be multiple options, e.g. a fall-through with state
transfer once a request cannot be handled yet or a sysctl with iptables
being the default handler and an option to switch to bpfilter for letting
it handle requests for that time being.
> 3. What happens if the rule can't be translated (yet?)
(See above.)
> 4. Do you plan to reimplement connection tracking in userspace?
One option could be to have a generic, skb-less connection tracker in kernel
that can be reused from the various hooks it would need to handle, potentially
that would also be able to get offloaded into HW as another benefit coming
out from that.
> If no, how will the bpf program interact with it?
> [ same question applies to ipv6 exthdr traversal, ip defragmentation
> and the like ].
The v6 exthdr traversal could be realized natively via BPF which should
make the parsing more robust at the same time than having it somewhere
inside a helper in kernel directly; bounded loops in BPF would help as
well on that front, similarly for defrag this could be handled by the prog
although here we would need additional infra to queue the packets and then
recirculate.
> I will probably have a quadrillion of followup questions, sorry :-/
Definitely, please do!
Thanks,
Daniel
>> Also, such architecture makes the kernel/user boundary very precise,
>> meaning requests can be handled and BPF translated in control plane part
>> in user space with its own user memory etc, while minimal data plane
>> bits are in kernel. It would also allow to remove old xtables modules
>> at some point from the kernel while keeping functionality in place.
>
> This is what we tried with nftables :-/
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 16:14 ` Florian Westphal
@ 2018-02-16 20:44 ` Daniel Borkmann
2018-02-17 12:33 ` Harald Welte
2018-02-17 19:18 ` Florian Westphal
2018-02-16 22:33 ` David Miller
1 sibling, 2 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-16 20:44 UTC (permalink / raw)
To: Florian Westphal; +Cc: netdev, netfilter-devel, davem, alexei.starovoitov
Hi Florian,
On 02/16/2018 05:14 PM, Florian Westphal wrote:
> Florian Westphal <fw@strlen.de> wrote:
>> Daniel Borkmann <daniel@iogearbox.net> wrote:
>> Several questions spinning at the moment, I will probably come up with
>> more:
>
> ... and here there are some more ...
>
> One of the many pain points of xtables design is the assumption of 'used
> only by sysadmin'.
>
> This has not been true for a very long time, so by now iptables has
> this userspace lock (yes, its fugly workaround) to serialize concurrent
> iptables invocations in userspace.
>
> AFAIU the translate-in-userspace design now brings back the old problem
> of different tools overwriting each others iptables rules.
Right, so the behavior would need to be adapted to be exactly the same,
given all the requests go into kernel space first via the usual uapis,
I don't think there would be anything in the way of keeping that as is.
> Another question -- am i correct in that each rule manipulation would
> incur a 'recompilation'? Or are there different mini programs chained
> together?
Right now in the PoC yes, basically it regenerates the program on the fly
in gen.c when walking the struct bpfilter_ipt_ip's and appends the entries
to the program, but it doesn't have to be that way. There are multiple
options to allow for a partial code generation, e.g. via chaining tail
call arrays or directly via BPF to BPF calls eventually, there would be
few changes on BPF side needed, but it can be done; there could additionally
be various optimizations passes during code generation phase performed
while keeping given constraints in order to speed up getting to a verdict.
> One of the nftables advantages is that (since rule representation in
> kernel is black-box from userspace point of view) is that the kernel
> can announce add/delete of rules or elements from nftables sets.
>
> Any particular reason why translating iptables rather than nftables
> (it should be possible to monitor the nftables changes that are
> announced by kernel and act on those)?
Yeah, correct, this should be possible as well. We started out with the
iptables part in the demo as the majority of bigger infrastructure projects
all still rely heavily on it (e.g. docker, k8s to just name two big ones).
Usually they have their requests to iptables baked into their code directly
which probably won't change any time soon, so thought was that they could
benefit initially from it once there would be sufficient coverage.
Thanks,
Daniel
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 14:57 ` [PATCH RFC 0/4] net: add bpfilter Florian Westphal
2018-02-16 16:14 ` Florian Westphal
2018-02-16 16:53 ` Daniel Borkmann
@ 2018-02-16 22:32 ` David Miller
2 siblings, 0 replies; 73+ messages in thread
From: David Miller @ 2018-02-16 22:32 UTC (permalink / raw)
To: fw; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
From: Florian Westphal <fw@strlen.de>
Date: Fri, 16 Feb 2018 15:57:27 +0100
> 4. Do you plan to reimplement connection tracking in userspace?
> If no, how will the bpf program interact with it?
The natural way to handle this, as with anything BPF related, is with
appropriate BPF helpers which would be added for this purpose.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 16:14 ` Florian Westphal
2018-02-16 20:44 ` Daniel Borkmann
@ 2018-02-16 22:33 ` David Miller
2018-02-17 12:21 ` Harald Welte
2018-02-17 20:10 ` Florian Westphal
1 sibling, 2 replies; 73+ messages in thread
From: David Miller @ 2018-02-16 22:33 UTC (permalink / raw)
To: fw; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
From: Florian Westphal <fw@strlen.de>
Date: Fri, 16 Feb 2018 17:14:08 +0100
> Any particular reason why translating iptables rather than nftables
> (it should be possible to monitor the nftables changes that are
> announced by kernel and act on those)?
As Daniel said, iptables is by far the most deployed of the two
technologies. Therefore it provides the largest environment for
testing and coverage.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
` (4 preceding siblings ...)
2018-02-16 14:57 ` [PATCH RFC 0/4] net: add bpfilter Florian Westphal
@ 2018-02-17 12:11 ` Harald Welte
2018-02-18 0:35 ` Florian Westphal
` (2 more replies)
2018-02-18 23:35 ` Florian Westphal
6 siblings, 3 replies; 73+ messages in thread
From: Harald Welte @ 2018-02-17 12:11 UTC (permalink / raw)
To: Daniel Borkmann; +Cc: netdev, netfilter-devel, davem, alexei.starovoitov
Hi Daniel,
On Fri, Feb 16, 2018 at 02:40:19PM +0100, Daniel Borkmann wrote:
> This is a very rough and early proof of concept that implements bpfilter.
> The basic idea of bpfilter is that it can process iptables queries and
> translate them in user space into BPF programs which can then get attached
> at various locations.
Interesting approach. My first question would be what the goal of all of this
is. For sure, one can implement many different things, but what is the use
case, and why do it this way?
I see several possible areas of contention:
1) If you aim for a non-feature-complete support of iptables rules, it
will create confusion to the users. When users use "iptables", they have
assumptions on what it will do and how it will behave. One can of course
replace / refactor the internal implementation, if the resulting behavior
is identical. And that means rules are executed at the same hooks in the stack,
with functionally identical matches and targets, provide the same
counter semantics, etc. But if the behavior is different, and/or the
provided functionality is different, then why "hide" this new
filtering technology behind iptables, rather than its own command
line tool? Such an alternative tool could share the same command
line syntax as iptables, or even provide a converter/wrapper, but
given that it would not be called "iptables" people will implicitly
have different assumptions about it
2) Why try to provide compatibility to iptables, when at the same time
many people have already migrated to (or are in the process of
migrating) to nftables? By using iptables semantics, structures,
architecture, you risk perpetuating the design mistakes we made in
iptables some 18 years ago for another decade or more. From my POV,
if one was to do eBPF optimized rule execution, it should be based on
nftables rather than iptables. This way you avoid the many
architectural problems, such as
* no incremental rule changes but only atomic swap of an entire table
with all its chains
* no common/shared rulesets for IPv4 + IPv6, which is very clumsy and
often worked around with ugly shellscript wrappers in userspace
which then call both iptables and ip6tables to add a rule to both
rulesets.
> The user space iptables binary issuing rule addition or dumps was
> left as-is, thus at some point any binaries against iptables uapi kernel
> interface could transparently be supported in such manner in long term.
See my comments above: In the netfilter community, we know for at least
a decade or more about the many problems of the old iptables userspace
interface. For many years, a much better replacement has been designed
as part of nftables.
> As rule translation can potentially become very complex, this is performed
> entirely in user space. In order to ease deployment, request_module() code
> is extended to allow user mode helpers to be invoked. Idea is that user mode
> helpers are built as part of the kernel build and installed as traditional
> kernel modules with .ko file extension into distro specified location,
> such that from a distribution point of view, they are no different than
> regular kernel modules.
That just blew my mind, sorry :) This goes much beyond
netfilter/iptables, and adds some quiet singificant new piece of
kernel/userspace infrastructure. To me, my apologies, it just sounds
like a quite strange hack. But then, I may lack the vision of how this
might be useful in other contexts.
I'm trying to understand why exactly one would
* use a 18 year old iptables userspace program with its equally old
setsockopt based interface between kernel and userspace
* insert an entire table with many chains of rules into the kernel
* re-eject that ruleset into another userspace program which then
compiles it into an eBPF program
* inserert that back into the kernel
To me, this looks like some kind of legacy backwards compatibility
mechanism that one would find in proprietary operating systems, but not
in Linux. iptables, libiptc etc. are all free software. The source
code can be edited, and you could just as well have a new version of
iptables and/or libiptc which would pass the ruleset in userspace to
your compiler, which would then insert the resulting eBPF program.
You could even have a LD_PRELOAD wrapper doing the same. That one
would even work with direct users of the iptables setsockopt inteerface.
Why add quite comprehensive kerne infrastructure? What's the motivation
here?
> Thus, allow request_module() logic to load such
> user mode helper (umh) binaries via:
>
> request_module("foo") ->
> call_umh("modprobe foo") ->
> sys_finit_module(FD of /lib/modules/.../foo.ko) ->
> call_umh(struct file)
>
> Such approach enables kernel to delegate functionality traditionally done
> by kernel modules into user space processes (either root or !root) and
> reduces security attack surface of such new code, meaning in case of
> potential bugs only the umh would crash but not the kernel. Another
> advantage coming with that would be that bpfilter.ko can be debugged and
> tested out of user space as well (e.g. opening the possibility to run
> all clang sanitizers, fuzzers or test suites for checking translation).
> Also, such architecture makes the kernel/user boundary very precise,
> meaning requests can be handled and BPF translated in control plane part
> in user space with its own user memory etc, while minimal data plane
> bits are in kernel.
I understand that it has advantages to have the compiler in userspace.
But then, why first send your rules into the kernel and back?
> In the implemented proof of concept we show that simple /32 src/dst IPs
> are translated in such manner.
Of course this is the first that one starts with. However, as we all
know, iptables was never very good or efficient about 5-tuple matching.
If you want a fast implementation of this, you don't use iptables which
does linear list iteration. The reason/rationale/use-case of iptables
is its many (I believe more than 100 now?) extensions both on the area
of matches and targets.
Some of those can be implemented easily in BPF (like recomputing the
checksum or the like). Some others I would find much more difficult -
particularly if you want to off-load it to the NIC. They require access
to state that only the kernel has (like 'cgroup' or 'owner' matching).
> In the below example, we show that dumping, loading and offloading of
> one or multiple simple rules work, we show the bpftool XDP dump of the
> generated BPF instruction sequence as well as a simple functional ping
> test to enforce policy in such way.
Could you please clarify why the 'filter' table INPUT chain was used if
you're using XDP? AFAICT they have completely different semantics.
There is a well-conceived and generally understood notion of where
exactly the filter/INPUT table processing happens. And that's not as
early as in the NIC, but it's much later in the processing of the
packet.
I believe _if_ one wants to use the approach of "hiding" eBPF behind
iptables, then either
a) the eBPF programs must be executed at the exact same points in the
stack as the existing hooks of the built-in chains of the
filter/nat/mangle/raw tables, or
b) you must introduce new 'tables', like an 'xdp' table which then has
the notion of processing very early in processing, way before the
normal filter table INPUT processing happens.
> Feedback very welcome!
Thanks. Despite being a former netfilter core team member, I'm trying
to look at this as neutral as possible. So please don't perceive my
comments as overly defensive or the like.
My main points are:
1) What is the goal of this?
2) Why iptables and not nftables?
3) If something looks like existing iptables, it must behave *exactly*
like existing iptables, otherwise it is prone to break users security
in subtle and very dangerous ways.
Looking forward to the following discussion and on other points of view.
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 22:33 ` David Miller
@ 2018-02-17 12:21 ` Harald Welte
2018-02-17 20:10 ` Florian Westphal
1 sibling, 0 replies; 73+ messages in thread
From: Harald Welte @ 2018-02-17 12:21 UTC (permalink / raw)
To: David Miller; +Cc: fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Fri, Feb 16, 2018 at 05:33:54PM -0500, David Miller wrote:
> From: Florian Westphal <fw@strlen.de>
>
> > Any particular reason why translating iptables rather than nftables
> > (it should be possible to monitor the nftables changes that are
> > announced by kernel and act on those)?
>
> As Daniel said, iptables is by far the most deployed of the two
> technologies. Therefore it provides the largest environment for
> testing and coverage.
As I outlined earlier, this way you are perpetuating the architectural
mistakes and constraints that were created ~ 18 years ago without any
benefit from the lessons learned ever since. In netfilter, we already
wanted to replace it as early as 2006 (AFAIR) with nfnetlink based
pkttables (which never materialized).
I would strongly suggest to focus on nftables (or even some other way of
configuration / userspace interaction) to ensure that the iptables
userspace interface can at some point be phased out eventually. Like we
did with ipchains before, and before that with ipfwadm.
By making a new implementation dependant on the oldest interface you are
perpetuating it. Sure, one can go that way, but I would suggest this to
be a *very* carefully weighed decision after a detailed
analysis/discusison.
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 20:44 ` Daniel Borkmann
@ 2018-02-17 12:33 ` Harald Welte
2018-02-17 19:18 ` Florian Westphal
1 sibling, 0 replies; 73+ messages in thread
From: Harald Welte @ 2018-02-17 12:33 UTC (permalink / raw)
To: Daniel Borkmann
Cc: Florian Westphal, netdev, netfilter-devel, davem, alexei.starovoitov
Hi Daniel,
On Fri, Feb 16, 2018 at 09:44:01PM +0100, Daniel Borkmann wrote:
> We started out with the
> iptables part in the demo as the majority of bigger infrastructure projects
> all still rely heavily on it (e.g. docker, k8s to just name two big ones).
docker is exec'ing the iptables command line program. So one could simply
offer a syntactically compatible userspace replacement that does the compilation
in userspce and avoid the iptables->libiptc->setsockopt->userspace roundtrip
and the associated changes to the kernel module loader you introduced.
kubernetes is using iptables-restore, which is part of iptables and
again has the same syntax. However, it aovids the per-rule fork+exec
overhead, which is why the netfilter project has been recommending it to
be used in such situations.
Do you have a list of known projects that use the legacy sockopt-based
iptables uapi directly, without using code from the iptables.git
codebase (e.g. libiptc, iptables or iptables-restore)? IMHO only
those projects would benefit from the approach you have taken vs. an
approach that simply offers a compatible commandline syntax.
> Usually they have their requests to iptables baked into their code directly
> which probably won't change any time soon, so thought was that they could
> benefit initially from it once there would be sufficient coverage.
If the binary offeers the same syntax (it could even be a fork/version
of the iptables codebase, only using the parsing without the existing
backend generating the ruleS), the same goal could be achieved.
The above of course assumes that you have a 100% functional replacement
(for 100% of the features that your use cases use) underneath the
"iptables command syntax" compatibility. But you need that in both
cases, whether you use the existing userspace api or not.
Regards,
Harald
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 20:44 ` Daniel Borkmann
2018-02-17 12:33 ` Harald Welte
@ 2018-02-17 19:18 ` Florian Westphal
1 sibling, 0 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-17 19:18 UTC (permalink / raw)
To: Daniel Borkmann
Cc: Florian Westphal, netdev, netfilter-devel, davem, alexei.starovoitov
Daniel Borkmann <daniel@iogearbox.net> wrote:
> Hi Florian,
>
> On 02/16/2018 05:14 PM, Florian Westphal wrote:
> > Florian Westphal <fw@strlen.de> wrote:
> >> Daniel Borkmann <daniel@iogearbox.net> wrote:
> >> Several questions spinning at the moment, I will probably come up with
> >> more:
> >
> > ... and here there are some more ...
> >
> > One of the many pain points of xtables design is the assumption of 'used
> > only by sysadmin'.
> >
> > This has not been true for a very long time, so by now iptables has
> > this userspace lock (yes, its fugly workaround) to serialize concurrent
> > iptables invocations in userspace.
> >
> > AFAIU the translate-in-userspace design now brings back the old problem
> > of different tools overwriting each others iptables rules.
>
> Right, so the behavior would need to be adapted to be exactly the same,
> given all the requests go into kernel space first via the usual uapis,
> I don't think there would be anything in the way of keeping that as is.
Uff. This isn't solveable. At least thats what I tried to say here.
This is a limitation of the xtables setsockopt interface design.
If $docker (or anything else) adds a new rule using plain iptables other
daemons are not aware of it.
If some deletes a rule added by $software it won't learn that either.
The "solutions" in place now (periodic reloads/'is my rule still in
place' etc. are not desirable long-term.
You'll also need 4 decoders for arp/ip/ip6/ebtables plus translations
for all matches and targets xtables currently has. (almost 100 i would
guess from quick glance).
Some of the more crazy ones also have external user visible interfaces
outside setsockopt (proc files, ipset).
> > One of the nftables advantages is that (since rule representation in
> > kernel is black-box from userspace point of view) is that the kernel
> > can announce add/delete of rules or elements from nftables sets.
> >
> > Any particular reason why translating iptables rather than nftables
> > (it should be possible to monitor the nftables changes that are
> > announced by kernel and act on those)?
>
> Yeah, correct, this should be possible as well. We started out with the
> iptables part in the demo as the majority of bigger infrastructure projects
> all still rely heavily on it (e.g. docker, k8s to just name two big ones).
Yes, which is why we have translation tools in place.
Just for the fun of it I tried to delete ip/ip6tables binaries on my
fedora27 laptop and replaced them with symlinks to
'xtables-compat-multi'.
Aside from two issues (SELinux denying 'iptables' to use netlink) and
one translation issue (-m rpfilter, which can be translated in current
upstream version) this works out of the box, the translator uses
nftables api to kernel (so kernel doesn't even know which program is
talking...), 'nft monitor' displays the rules being added, and
'nft list ruleset' shows the default firewalld ruleset.
Obviously there are a few limitations, for instance ip6tables-save will
stop working once you add nft-based rules that use features that cannot
be expressed in xtables syntax (it will throw an error message similar
to 'you are using nftables featues not available in xtables, please use
nft'), for intance verdict maps, sets and the like.
> Usually they have their requests to iptables baked into their code directly
> which probably won't change any time soon, so thought was that they could
> benefit initially from it once there would be sufficient coverage.
See above, the translator covers most basic use cases nowadays.
The more extreme cases are not covered because we were reluctant to
provide equivalent in nftables (-m time comes to mind which was always a
PITA because kernel has no notion of timezone or DST transitions,
leading to 'magic' mismatches when timezone changes...
I could explain on more problem cases but none of them are too
important I think.
If you'd like to have more ebpf users in the kernel, then there is at
least one use case where ebpf could be very attractive for nftables
(matching dynamic headers and the like). This would be a new
feature and would need changes on nftables userspace side
as well (we don't have syntax/grammar to represent this in either
nft or iptables).
In most basic form, it would be nftables replacement for '-m string'
(and perhaps also -m bpf to some degree, depends on how it would be
realized).
We can discuss more if there is interest, but I think it
would be more suitable for conference/face to face discussion.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 22:33 ` David Miller
2018-02-17 12:21 ` Harald Welte
@ 2018-02-17 20:10 ` Florian Westphal
2018-02-17 22:38 ` Florian Westphal
1 sibling, 1 reply; 73+ messages in thread
From: Florian Westphal @ 2018-02-17 20:10 UTC (permalink / raw)
To: David Miller; +Cc: fw, daniel, netdev, netfilter-devel, alexei.starovoitov
David Miller <davem@davemloft.net> wrote:
> From: Florian Westphal <fw@strlen.de>
> Date: Fri, 16 Feb 2018 17:14:08 +0100
>
> > Any particular reason why translating iptables rather than nftables
> > (it should be possible to monitor the nftables changes that are
> > announced by kernel and act on those)?
>
> As Daniel said, iptables is by far the most deployed of the two
> technologies. Therefore it provides the largest environment for
> testing and coverage.
Right, but the approach of hooking old blob format comes with
lots of limitations that were meant to be resolved with a netlink based
interface which places kernel in a position to mediate all transactions
to the rule database (which isn't fixable with old setsockopt format).
As all programs call iptables(-restore) or variants translation can
be done in userspace to nftables so api spoken is nfnetlink.
Such a translator already exists and can handle some cases already:
nft flush ruleset
nft list ruleset | wc -l
0
xtables-compat-multi iptables -A INPUT -s 192.168.0.24 -j ACCEPT
xtables-compat-multi iptables -A INPUT -s 192.168.0.0/16 -p tcp --dport 22 -j ACCEPT
xtables-compat-multi iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
xtables-compat-multi iptables -A INPUT -p icmp -j ACCEPT
xtables-compat-multi iptables -N REJECT_LOG
xtables-compat-multi iptables -A REJECT_LOG -i eth0 -p tcp --tcp-flags SYN,ACK SYN --dport 22:80 -m limit --limit 1/sec -j LOG --log-prefix "RejectTCPConnectReq"
xtables-compat-multi iptables -A REJECT_LOG -j DROP
xtables-compat-multi iptables -A INPUT -j REJECT_LOG
nft list ruleset
table ip filter {
chain INPUT {
type filter hook input priority 0; policy accept;
ip saddr 192.168.0.24 counter packets 0 bytes 0 accept
ip saddr 192.168.0.0/16 tcp dport 22 counter accept
iifname "eth0" ct state related,established counter accept
ip protocol icmp counter packets 0 bytes 0 accept
counter packets 0 bytes 0 jump REJECT_LOG
}
chain FORWARD {
type filter hook forward priority 0; policy accept;
}
chain OUTPUT {
type filter hook output priority 0; policy accept;
}
chain REJECT_LOG {
iifname "eth0" tcp dport 22-80 tcp flags & (syn | ack) == syn limit rate 1/second burst 5 packets counter packets 0 bytes 0 log prefix "RejectTCPConnectReq"
counter packets 0 bytes 0 drop
}
}
and, while 'iptables' rules were added, nft monitor in different terminal:
nft monitor
add table ip filter
add chain ip filter INPUT { type filter hook input priority 0; policy accept; }
add chain ip filter FORWARD { type filter hook forward priority 0; policy accept; }
add chain ip filter OUTPUT { type filter hook output priority 0; policy accept; }
add rule ip filter INPUT ip saddr 192.168.0.24 counter packets 0 bytes 0 accept
# new generation 9893 by process 7471 (xtables-compat-)
add rule ip filter INPUT ip saddr 192.168.0.0/16 tcp dport 22 counter accept
# new generation 9894 by process 7504 (xtables-compat-)
add rule ip filter INPUT iifname "eth0" ct state related,established counter accept
# new generation 9895 by process 7528 (xtables-compat-)
add rule ip filter INPUT ip protocol icmp counter packets 0 bytes 0 accept
# new generation 9896 by process 7542 (xtables-compat-)
add chain ip filter REJECT_LOG
# new generation 9897 by process 7595 (xtables-compat-)
add rule ip filter REJECT_LOG iifname "eth0" tcp dport 22-80 tcp flags & (syn | ack) == syn limit rate 1/second burst 5 packets counter packets 0 bytes 0 log prefix "RejectTCPConnectReq"
# new generation 9898 by process 7639 (xtables-compat-)
add rule ip filter REJECT_LOG counter packets 0 bytes 0 drop
# new generation 9899 by process 7657 (xtables-compat-)
add rule ip filter INPUT counter packets 0 bytes 0 jump REJECT_LOG
# new generation 9900 by process 7663 (xtables-compat-)
Now, does this work in all cases?
Unfortunately not -- this is still work-in-progress, so I would
not rm /sbin/iptables and replace it with a link to xtables-compat-multi just yet.
(f.e. nftables misses some selinux matches/targets for netlabel so we obviously
can't translate this, same for ipsec sa/policy matching -- but this isn't
impossible to resolve).
Hopefully this does show that at least some commonly used features work
and that we've come a long way to make seamless nftables transition happen.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-17 20:10 ` Florian Westphal
@ 2018-02-17 22:38 ` Florian Westphal
0 siblings, 0 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-17 22:38 UTC (permalink / raw)
To: Florian Westphal
Cc: David Miller, daniel, netdev, netfilter-devel, alexei.starovoitov
Florian Westphal <fw@strlen.de> wrote:
> David Miller <davem@davemloft.net> wrote:
> > From: Florian Westphal <fw@strlen.de>
> > Date: Fri, 16 Feb 2018 17:14:08 +0100
> >
> > > Any particular reason why translating iptables rather than nftables
> > > (it should be possible to monitor the nftables changes that are
> > > announced by kernel and act on those)?
> >
> > As Daniel said, iptables is by far the most deployed of the two
> > technologies. Therefore it provides the largest environment for
> > testing and coverage.
>
> Right, but the approach of hooking old blob format comes with
> lots of limitations that were meant to be resolved with a netlink based
> interface which places kernel in a position to mediate all transactions
> to the rule database (which isn't fixable with old setsockopt format).
>
> As all programs call iptables(-restore) or variants translation can
> be done in userspace to nftables so api spoken is nfnetlink.
> Such a translator already exists and can handle some cases already:
>
> nft flush ruleset
> nft list ruleset | wc -l
> 0
> xtables-compat-multi iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
> xtables-compat-multi iptables -A REJECT_LOG -i eth0 -p tcp --tcp-flags SYN,ACK SYN --dport 22:80 -m limit --limit 1/sec -j LOG --log-prefix "RejectTCPConnectReq"
to be fair, for these two I had to use
$(xtables-compat-multi iptables-translate -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT)
Reason is that the 'iptables-translate' part nowadays has way more
translations available (nft gained many features since the
iptables-compat layer was added).
If given appropriate prioriy however it should be pretty
trivial to make the 'translate' descriptions available in
the 'direct' version, we already have function in libnftables
to execute/run a command directly from a buffer so this would
not even need fork/execve overhead (although I don't think
its a big concern).
> (f.e. nftables misses some selinux matches/targets for netlabel so we obviously
> can't translate this, same for ipsec sa/policy matching -- but this isn't
> impossible to resolve).
I am working on some poc code for the sa/policy thing now.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-17 12:11 ` Harald Welte
@ 2018-02-18 0:35 ` Florian Westphal
2018-02-19 12:03 ` Daniel Borkmann
2018-02-19 21:42 ` Willem de Bruijn
2 siblings, 0 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-18 0:35 UTC (permalink / raw)
To: Harald Welte
Cc: Daniel Borkmann, netdev, netfilter-devel, davem, alexei.starovoitov
Harald Welte <laforge@gnumonks.org> wrote:
> I believe _if_ one wants to use the approach of "hiding" eBPF behind
> iptables, then either
[..]
> b) you must introduce new 'tables', like an 'xdp' table which then has
> the notion of processing very early in processing, way before the
> normal filter table INPUT processing happens.
In nftables. the netdev ingress hook location could be used for this,
but right, iptables has no equivalent.
netdev ingress is interesting from an hw-offload point of view,
unlike all other netfilter hooks its tied to a specific network interface
rather than owned by the network namespace.
A rule like (yes i am making this up)
limit 10000 byte/s
cannot be offloaded because it affects all packets going through
the system, i.e. you'd need to share state among all nics which i think
won't work :-)
Same goes for any other match/target that somehow contains (global)
state and was added to the 'classic' iptables hook points.
(exception: rule restricts interface via '-i foo').
Note well: "offloaded != ebpf" in this case.
I see no reasons why ebpf cannot be used in either iptables or
nftables. How to get there is obviously a different beast.
For iptables, I think we should put it in maintenance mode and
focus on nftables, for many reasons outlined in other replies.
And how to best make use of ebpf+nftables
In ideal world, nftables would have used (e)bpf from the start.
But, well, its not an ideal world (iirc nft origins are just a bit
too old).
That doesn't mean that we can't leverage ebpf from nftables.
Its just a question of where it makes sense and where it doesn't,
f.e. i see no reason to replace c code with ebpf just 'because you can'.
Speedup? Good argument.
Feature enhancements that could use ebpf programs? Another good
argument.
I guess there are a lot more.
So I'd like to second Haralds question.
What is the main goal?
For nftables, I believe most important ones are:
- make kernel keeper/owner of all rules
- allow userspace to learn of rule addition/deletion
- provide fast matching (no linear evaluation of rules,
native sets with jump and verdict maps)
- provide a single tool instead of ip/ip6/arp/ebtables
- unified ipv4/ipv6 matching
- backwards compat and/or translation infrastructure
But once these are reached, we will hopefully have more:
- offloading (hardware)
- speedup via JIT compilation
- feature enhancements such as matching arbitrary packet
contents
I suspect you see that ebpf might be a fit and/or help us with
all of these things.
So, once I understand what your goals are I might be better able
to see how nftables could fit into the picture, as you can see
I did a lot of guesswork :-)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
` (5 preceding siblings ...)
2018-02-17 12:11 ` Harald Welte
@ 2018-02-18 23:35 ` Florian Westphal
6 siblings, 0 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-18 23:35 UTC (permalink / raw)
To: Daniel Borkmann; +Cc: netdev, netfilter-devel, davem, alexei.starovoitov
Daniel Borkmann <daniel@iogearbox.net> wrote:
> As rule translation can potentially become very complex, this is performed
> entirely in user space. In order to ease deployment, request_module() code
> is extended to allow user mode helpers to be invoked. Idea is that user mode
> helpers are built as part of the kernel build and installed as traditional
> kernel modules with .ko file extension into distro specified location,
> such that from a distribution point of view, they are no different than
> regular kernel modules. Thus, allow request_module() logic to load such
> user mode helper (umh) binaries via:
>
> request_module("foo") ->
> call_umh("modprobe foo") ->
> sys_finit_module(FD of /lib/modules/.../foo.ko) ->
> call_umh(struct file)
>
> Such approach enables kernel to delegate functionality traditionally done
> by kernel modules into user space processes (either root or !root)
Unrelated: AFAIU this would allow to e.g. move the compat32 handlers
(which are very ugly/error prone) off to userspace?
compat_syscall -> umh_32_64_xlate -> syscall() ?
[ feel free to move this to different thread, only mentioning this
so I won't forget ]
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-17 12:11 ` Harald Welte
2018-02-18 0:35 ` Florian Westphal
@ 2018-02-19 12:03 ` Daniel Borkmann
2018-02-19 12:52 ` Harald Welte
2018-02-19 15:00 ` David Miller
2018-02-19 21:42 ` Willem de Bruijn
2 siblings, 2 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-19 12:03 UTC (permalink / raw)
To: Harald Welte; +Cc: netdev, netfilter-devel, davem, alexei.starovoitov
Hi Harald,
On 02/17/2018 01:11 PM, Harald Welte wrote:
[...]
>> As rule translation can potentially become very complex, this is performed
>> entirely in user space. In order to ease deployment, request_module() code
>> is extended to allow user mode helpers to be invoked. Idea is that user mode
>> helpers are built as part of the kernel build and installed as traditional
>> kernel modules with .ko file extension into distro specified location,
>> such that from a distribution point of view, they are no different than
>> regular kernel modules.
>
> That just blew my mind, sorry :) This goes much beyond
> netfilter/iptables, and adds some quiet singificant new piece of
> kernel/userspace infrastructure. To me, my apologies, it just sounds
> like a quite strange hack. But then, I may lack the vision of how this
> might be useful in other contexts.
Thought was that it would be more suitable to push all the complexity of
such translation into user space which brings couple of additional advantages
as well: the translation can become very complex and thus it would contain
all of it behind syscall boundary where natural path of loading programs
would go via verifier. Given the tool would reside in user space, it would
also allow to ease development and testing can happen w/o recompiling the
kernel. It would allow for all the clang sanitizers to run there and for
having a comprehensive test suite to verify and dry test translations against
traffic test patterns (e.g. bpf infra would provide possibilities on this
w/o complex setup). Given normal user mode helpers make this rather painful
since they need to be shipped as extra package by the various distros, the
idea was that the module loader back end could treat umh similarly as kernel
modules and hook them in through request_module() approach while still
operating out of user space. In any case, I could image this approach might
be interesting and useful in general also for other subsystems requiring
umh in one way or another.
> I'm trying to understand why exactly one would
> * use a 18 year old iptables userspace program with its equally old
> setsockopt based interface between kernel and userspace
> * insert an entire table with many chains of rules into the kernel
> * re-eject that ruleset into another userspace program which then
> compiles it into an eBPF program
> * inserert that back into the kernel
>
> To me, this looks like some kind of legacy backwards compatibility
> mechanism that one would find in proprietary operating systems, but not
> in Linux. iptables, libiptc etc. are all free software. The source
> code can be edited, and you could just as well have a new version of
> iptables and/or libiptc which would pass the ruleset in userspace to
> your compiler, which would then insert the resulting eBPF program.
>
> You could even have a LD_PRELOAD wrapper doing the same. That one
> would even work with direct users of the iptables setsockopt inteerface.
>
> Why add quite comprehensive kerne infrastructure? What's the motivation
> here?
Right, having a custom iptables, libiptc or LD_PRELOAD approach would work
as well of course, but it still wouldn't address applications that have
their own custom libs programmed against iptables uapi directly or those
that reused a builtin or modified libiptc directly in their application.
Such requests could only be covered transparently by having a small shim
layer in kernel and it also wouldn't require any extra packages from distro
side.
[...]
>> In the implemented proof of concept we show that simple /32 src/dst IPs
>> are translated in such manner.
>
> Of course this is the first that one starts with. However, as we all
> know, iptables was never very good or efficient about 5-tuple matching.
> If you want a fast implementation of this, you don't use iptables which
> does linear list iteration. The reason/rationale/use-case of iptables
> is its many (I believe more than 100 now?) extensions both on the area
> of matches and targets.
>
> Some of those can be implemented easily in BPF (like recomputing the
> checksum or the like). Some others I would find much more difficult -
> particularly if you want to off-load it to the NIC. They require access
> to state that only the kernel has (like 'cgroup' or 'owner' matching).
Yeah, when it comes to offloading, the latter two examples are heavily tied
to upper layers of the (local) stack, so for cases like those it wouldn't
make much sense, but e.g. matches, mangling or forwarding based on packet
data are obvious candidates that can already be offloaded today in a
flexible and programmable manner all with existing BPF infra, so for those
it could definitely be highly interesting to make use of it.
>> In the below example, we show that dumping, loading and offloading of
>> one or multiple simple rules work, we show the bpftool XDP dump of the
>> generated BPF instruction sequence as well as a simple functional ping
>> test to enforce policy in such way.
>
> Could you please clarify why the 'filter' table INPUT chain was used if
> you're using XDP? AFAICT they have completely different semantics.
>
> There is a well-conceived and generally understood notion of where
> exactly the filter/INPUT table processing happens. And that's not as
> early as in the NIC, but it's much later in the processing of the
> packet.
>
> I believe _if_ one wants to use the approach of "hiding" eBPF behind
> iptables, then either
>
> a) the eBPF programs must be executed at the exact same points in the
> stack as the existing hooks of the built-in chains of the
> filter/nat/mangle/raw tables, or
>
> b) you must introduce new 'tables', like an 'xdp' table which then has
> the notion of processing very early in processing, way before the
> normal filter table INPUT processing happens.
The semantics would need to match the exact same points fully agree, so
wrt hooks the ingress one would have been a more close candidate for the
provided example.
With regards to why iptables and not nftables, definitely a legit question.
Right now most of the infra still relies on iptables in one way or another,
is there an answer for the existing applications that programmed against
uapi directly for the long term?
Meaning once the switch would be flipped to nftables entirely, would they
stop working when old iptables gets removed?
When would be the ETA for that?
Not opposed at all for some sort of translation for the latter, just wondering
since for the majority it would probably still take years to come to make
a migration.
Thanks,
Daniel
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 12:03 ` Daniel Borkmann
@ 2018-02-19 12:52 ` Harald Welte
2018-02-19 14:44 ` David Miller
2018-02-19 15:00 ` David Miller
1 sibling, 1 reply; 73+ messages in thread
From: Harald Welte @ 2018-02-19 12:52 UTC (permalink / raw)
To: Daniel Borkmann; +Cc: netdev, netfilter-devel, davem, alexei.starovoitov
Hi Daniel,
On Mon, Feb 19, 2018 at 01:03:17PM +0100, Daniel Borkmann wrote:
> Hi Harald,
>
> On 02/17/2018 01:11 PM, Harald Welte wrote:
> [...]
> >> As rule translation can potentially become very complex, this is performed
> >> entirely in user space. In order to ease deployment, request_module() code
> >> is extended to allow user mode helpers to be invoked. Idea is that user mode
> >> helpers are built as part of the kernel build and installed as traditional
> >> kernel modules with .ko file extension into distro specified location,
> >> such that from a distribution point of view, they are no different than
> >> regular kernel modules.
> >
> > That just blew my mind, sorry :) This goes much beyond
> > netfilter/iptables, and adds some quiet singificant new piece of
> > kernel/userspace infrastructure. To me, my apologies, it just sounds
> > like a quite strange hack. But then, I may lack the vision of how this
> > might be useful in other contexts.
>
> Thought was that it would be more suitable to push all the complexity of
> such translation into user space [...]
Sure, you have no complaints from my side about that goal. I'm just not
sure if turning the kernel module loader into a new mechanism to start
userspace processes is. I guess that's a question that the people
involved with core kernel code and module loader have to answer. To me
it seems like a very loooong detour away from the actual topic (packet
filtering).
> Given normal user mode helpers make this rather painful since they
> need to be shipped as extra package by the various distros, the idea
> was that the module loader back end could treat umh similarly as
> kernel modules and hook them in through request_module() approach
> while still operating out of user space. In any case, I could image
> this approach might be interesting and useful in general also for
> other subsystems requiring umh in one way or another.
I completely agree this approach has some logic to it. I just think the
approach taken is *very* different from what has been traditionally done
in the Linux world. All sorts of userspace programs to configure kernel
features (iptables being one of them iproute2, etc.) have always been
distributed as separate/independent application programs, which are
packaged separately, etc.
Making the kernel source tree build such userspace utilities and
executing them in a new fashion via the kernel module loaders are to me
two quite large conceptual changes on how "Linux works", and I believe
you will have to "sell" this idea to many people outside the kernel
networking communit, i.e. core kernel developers, people who do
packaging, etc.
I'm not saying I'm fundamentally opposed to it. Will be curious to see
how the wider kernel community thinks of that architecture.
> Right, having a custom iptables, libiptc or LD_PRELOAD approach would work
> as well of course, but it still wouldn't address applications that have
> their own custom libs programmed against iptables uapi directly or those
> that reused a builtin or modified libiptc directly in their application.
How many of those wide-spread applications are you aware of? The two
projects you have pointed out (docker and kubernetes) don't. As the
assumption that many such tools would need to be supported drives a lot
of the design decisions, I would argue one needs a solid empircal basis.
Also, the LD_PRELOAD wrapper *would* work with all those programs. Only
the iptables command line replacement wouldn't catch those.
> Such requests could only be covered transparently by having a small shim
> layer in kernel and it also wouldn't require any extra packages from distro
> side.
What is wrong with extra packages in distributions? Distributions also
will have to update the kernel to include your new code, so they could
at the same time use a new iptables (or $whatever) package. This is
true for virtually all new kernel features. Your userland needs to go
along with it, if it wants to use those new features.
> > Some of those can be implemented easily in BPF (like recomputing the
> > checksum or the like). Some others I would find much more difficult -
> > particularly if you want to off-load it to the NIC. They require access
> > to state that only the kernel has (like 'cgroup' or 'owner' matching).
>
> Yeah, when it comes to offloading, the latter two examples are heavily tied
> to upper layers of the (local) stack, so for cases like those it wouldn't
> make much sense, but e.g. matches, mangling or forwarding based on packet
> data are obvious candidates that can already be offloaded today in a
> flexible and programmable manner all with existing BPF infra, so for those
> it could definitely be highly interesting to make use of it.
While I believe you there are many ways how one can offload things
flexibly with eBPF, I still have a hard time understanding how you want
to merge this with the existing well-defined notion of when exactly a
given chain of a given table is executed.
* compatibility with iptables only makes sense if you use the legacy
filter/nat/raw/mangle tables and the existing points in the stack,
such as PRE_ROUTING/POST_ROUTING/LOCAL_IN/LOCAL_OUT
* offloading those to a NIC seems rather hard to me, as you basically
have already traversed half of the Linux kernel stack and then
suddenly need to feed it back into your smart-nic, only to pull it
back from there to continue traversing the Linux stack. Forgive me if
I'm not following recent developments closely enough if that problem
has already been solved.
* as soon as you define new 'netfilter hooks' to which the
ip/ip6/arp/... tables can bind, you can of course define such new
tables with new built-in chains. However, at that point I'm starting
to wonder why you would then want to go for "iptables compatibility"
in the first place, as no existing rulesets / programs can be used 1:1
as they need to design their rulesets based on those new
tables/hooks/chains
So conceptually, I think if you want semantic backwards compatibility,
all you can do is to translate all rules attached to a given netfilter
hook into a eBPF program, and then execute that program instead of the
normal ip_tables traversal. Adn that eBPF would then run on the normal
host CPU.
> The semantics would need to match the exact same points fully agree, so
> wrt hooks the ingress one would have been a more close candidate for the
> provided example.
>
> With regards to why iptables and not nftables, definitely a legit question.
> Right now most of the infra still relies on iptables in one way or another,
> is there an answer for the existing applications that programmed against
> uapi directly for the long term?
As Pablo and Florian have commented, there are legacy
translators/wrappers that "look and feel like iptables" but in tern
translate the rules to nftables and then use the nftables netlink
interface to manage the rules in the kernel. Is it a perfect 100.0%
translation? No. How close it is and what's missing is something they
have to comment on.
> Meaning once the switch would be flipped to nftables entirely, would they
> stop working when old iptables gets removed?
The various translation/migration scenarios are described here
https://wiki.nftables.org/wiki-nftables/index.php/Moving_from_iptables_to_nftables
> When would be the ETA for that?
I cannot comment on this. The current netfilter core team will have to
respond to this.
> Not opposed at all for some sort of translation for the latter, just wondering
> since for the majority it would probably still take years to come to make
> a migration.
Migrations take some time, sure. It will also take years if such a
iptables-to-ebpf translator development gets completed, included in
mainline, distributions enable it, roll out such kernel versions, ...
In general, I would argue that compatibility on the command line level
of {ip,ip6,arp}{tables,tables-{save,restore}} is the most important
level of compatibility. This is what has been done at previous
migrations e.g. ipchains->iptables before.
Translators already exist for *tables -> nftables. (possibly slightly dated) Status can be found at
https://wiki.nftables.org/wiki-nftables/index.php/List_of_available_translations_via_iptables-translate_tool
and
https://wiki.nftables.org/wiki-nftables/index.php/Supported_features_compared_to_xtables
It would be an interesting test to see if e.g. docker would run on top
of the translator. I have no idea if anyone has tried this. It would
for sure be an interesting investigation. I would much rather see
effort spent on improving the existing translators, or helping those
projects doing the switch to nftables (or any other new technology) than
to introduce new technology using 18-year-old uapi interfaces that we
all know have many problems.
Regards,
Harald
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 12:52 ` Harald Welte
@ 2018-02-19 14:44 ` David Miller
2018-02-19 14:53 ` Florian Westphal
2018-02-19 15:23 ` Harald Welte
0 siblings, 2 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 14:44 UTC (permalink / raw)
To: laforge; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
From: Harald Welte <laforge@gnumonks.org>
Date: Mon, 19 Feb 2018 13:52:18 +0100
>> Right, having a custom iptables, libiptc or LD_PRELOAD approach would work
>> as well of course, but it still wouldn't address applications that have
>> their own custom libs programmed against iptables uapi directly or those
>> that reused a builtin or modified libiptc directly in their application.
>
> How many of those wide-spread applications are you aware of? The two
> projects you have pointed out (docker and kubernetes) don't. As the
> assumption that many such tools would need to be supported drives a lot
> of the design decisions, I would argue one needs a solid empircal basis.
I see talk about "just replacing the iptables binary".
A long time ago in a galaxy far far away, that would be a reasonable
scheme. But that kind of approach won't work in the realities of
today.
You aren't going to be able to replace the iptables binary in the tens
of thousands of container images out there, nor the virtualization
installations either.
Like it or not iptables ABI based filtering is going to be in the data
path for many years if not a decade or more to come. iptables is a
victim of it's own success, like it or not :-) Yes, the ABI is
terrible, but obviously it was useful enough for lots of people.
Therefore it behooves us to accept this reality and align the data
path generated to match what the rest of the kernel is moving towards
and that is eBPF and XDP.
Furthrmore, on a long term maintainence perspective, it means that
every data path used by the kernel for iptables will be fully verified
by the eBPF verifier. This means that the iptables data path will be
guaranteed to never get into a loop, access out of bounds data, etc.
That to me is real power, and something we should pursue.
This doesn't even get into the offloading and other benefits that are
possible. I know you can't see how offloading is possible, but I hope
are some further discussion you can see how that might work.
Thanks.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 14:44 ` David Miller
@ 2018-02-19 14:53 ` Florian Westphal
2018-02-19 15:07 ` David Miller
2018-02-19 15:23 ` Harald Welte
1 sibling, 1 reply; 73+ messages in thread
From: Florian Westphal @ 2018-02-19 14:53 UTC (permalink / raw)
To: David Miller; +Cc: laforge, daniel, netdev, netfilter-devel, alexei.starovoitov
David Miller <davem@davemloft.net> wrote:
> > How many of those wide-spread applications are you aware of? The two
> > projects you have pointed out (docker and kubernetes) don't. As the
> > assumption that many such tools would need to be supported drives a lot
> > of the design decisions, I would argue one needs a solid empircal basis.
>
> I see talk about "just replacing the iptables binary".
>
> A long time ago in a galaxy far far away, that would be a reasonable
> scheme. But that kind of approach won't work in the realities of
> today.
>
> You aren't going to be able to replace the iptables binary in the tens
> of thousands of container images out there, nor the virtualization
> installations either.
Why would you have to?
iptables kernel parts are still maintained, its not dead code that
stands in the way.
We can leave it alone, in maintenance mode, just fine.
> Like it or not iptables ABI based filtering is going to be in the data
> path for many years if not a decade or more to come. iptables is a
> victim of it's own success, like it or not :-) Yes, the ABI is
> terrible, but obviously it was useful enough for lots of people.
Sure, but looking at all the things that were added to iptables
to alleviate some of the issues (ipset for instance) show that we need a
meaningful re-design of how things work conceptually.
The umh helper translation that has been proposed could be applied to
transparently xlate iptables to nftables (or e.g. iptables compat32 to
iptables64), i.e. legacy binary talks to kernel, kernel invokes umh, umh
generates nftables netlink messages).
But I don't even see a need to do this, I don't think its an issue
to leave it in the tree even for another decade or more if needed be.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:00 ` David Miller
@ 2018-02-19 14:59 ` Florian Westphal
2018-02-19 15:13 ` David Miller
0 siblings, 1 reply; 73+ messages in thread
From: Florian Westphal @ 2018-02-19 14:59 UTC (permalink / raw)
To: David Miller; +Cc: daniel, laforge, netdev, netfilter-devel, alexei.starovoitov
David Miller <davem@davemloft.net> wrote:
> From: Daniel Borkmann <daniel@iogearbox.net>
> Date: Mon, 19 Feb 2018 13:03:17 +0100
>
> > Thought was that it would be more suitable to push all the complexity of
> > such translation into user space which brings couple of additional advantages
> > as well: the translation can become very complex and thus it would contain
> > all of it behind syscall boundary where natural path of loading programs
> > would go via verifier. Given the tool would reside in user space, it would
> > also allow to ease development and testing can happen w/o recompiling the
> > kernel. It would allow for all the clang sanitizers to run there and for
> > having a comprehensive test suite to verify and dry test translations against
> > traffic test patterns (e.g. bpf infra would provide possibilities on this
> > w/o complex setup). Given normal user mode helpers make this rather painful
> > since they need to be shipped as extra package by the various distros, the
> > idea was that the module loader back end could treat umh similarly as kernel
> > modules and hook them in through request_module() approach while still
> > operating out of user space. In any case, I could image this approach might
> > be interesting and useful in general also for other subsystems requiring
> > umh in one way or another.
>
> Yes, this is a very powerful new facility.
>
> It also means that the scope of developers who can contribute and work
> on the translater is much larger.
How so? Translator is in userspace in nftables case too?
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 12:03 ` Daniel Borkmann
2018-02-19 12:52 ` Harald Welte
@ 2018-02-19 15:00 ` David Miller
2018-02-19 14:59 ` Florian Westphal
1 sibling, 1 reply; 73+ messages in thread
From: David Miller @ 2018-02-19 15:00 UTC (permalink / raw)
To: daniel; +Cc: laforge, netdev, netfilter-devel, alexei.starovoitov
From: Daniel Borkmann <daniel@iogearbox.net>
Date: Mon, 19 Feb 2018 13:03:17 +0100
> Thought was that it would be more suitable to push all the complexity of
> such translation into user space which brings couple of additional advantages
> as well: the translation can become very complex and thus it would contain
> all of it behind syscall boundary where natural path of loading programs
> would go via verifier. Given the tool would reside in user space, it would
> also allow to ease development and testing can happen w/o recompiling the
> kernel. It would allow for all the clang sanitizers to run there and for
> having a comprehensive test suite to verify and dry test translations against
> traffic test patterns (e.g. bpf infra would provide possibilities on this
> w/o complex setup). Given normal user mode helpers make this rather painful
> since they need to be shipped as extra package by the various distros, the
> idea was that the module loader back end could treat umh similarly as kernel
> modules and hook them in through request_module() approach while still
> operating out of user space. In any case, I could image this approach might
> be interesting and useful in general also for other subsystems requiring
> umh in one way or another.
Yes, this is a very powerful new facility.
It also means that the scope of developers who can contribute and work
on the translater is much larger.
When we showed this infrastructure to Linus he thought it was a very
sane idea.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 14:53 ` Florian Westphal
@ 2018-02-19 15:07 ` David Miller
2018-02-19 15:20 ` Florian Westphal
0 siblings, 1 reply; 73+ messages in thread
From: David Miller @ 2018-02-19 15:07 UTC (permalink / raw)
To: fw; +Cc: laforge, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Florian Westphal <fw@strlen.de>
Date: Mon, 19 Feb 2018 15:53:14 +0100
> Sure, but looking at all the things that were added to iptables
> to alleviate some of the issues (ipset for instance) show that we need a
> meaningful re-design of how things work conceptually.
As you said iptables is in maintainenance mode.
But there are millions upon millions of users, like it or not, and
they aren't going away for decades. And this is the iptables binary
ABI I'm talking about, not the iptables user command line interface.
These discussions about nftables migrations sound like a person near a
power outage who exclaims: "What's the big deal, the lights are on in
my house?" Please see further than the view inside your home.
By in large, we are stuck with iptables's data path for an extremely
long time.
Major data centers doesn't even enable NFTABLES in their kernels, and
there is nothing you can do about that in the short to medium term.
Therefore, for all of the beneficial reasons I have discussed we
should make that datapath as aligned and integrated with our core
important technologies as possible, so that they can benefit from any
and all improvements in that area rather than just collecting dust.
Thank you.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 14:59 ` Florian Westphal
@ 2018-02-19 15:13 ` David Miller
2018-02-19 15:15 ` Florian Westphal
2018-02-19 15:27 ` Harald Welte
0 siblings, 2 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 15:13 UTC (permalink / raw)
To: fw; +Cc: daniel, laforge, netdev, netfilter-devel, alexei.starovoitov
From: Florian Westphal <fw@strlen.de>
Date: Mon, 19 Feb 2018 15:59:35 +0100
> David Miller <davem@davemloft.net> wrote:
>> It also means that the scope of developers who can contribute and work
>> on the translater is much larger.
>
> How so? Translator is in userspace in nftables case too?
Florian, first of all, the whole "change the iptables binary" idea is
a non-starter. For the many reasons I have described in the various
postings I have made today.
It is entirely impractical.
So we are strictly talking about the code we are writing to translate
iptables ABI (in the kernel) into an eBPF based datapath.
Anything designed in that nature must be distributed completely in the
kernel tree, so that the iptables kernel ABI is provided without any
externel dependencies.
We could have done the translater in in the kernel, but instead we are
doing it with a userland component.
And that's what we are talking about.
Thank you.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:13 ` David Miller
@ 2018-02-19 15:15 ` Florian Westphal
2018-02-19 15:27 ` David Miller
2018-02-19 21:30 ` Jozsef Kadlecsik
2018-02-19 15:27 ` Harald Welte
1 sibling, 2 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-19 15:15 UTC (permalink / raw)
To: David Miller
Cc: fw, daniel, laforge, netdev, netfilter-devel, alexei.starovoitov
David Miller <davem@davemloft.net> wrote:
> From: Florian Westphal <fw@strlen.de>
> Date: Mon, 19 Feb 2018 15:59:35 +0100
>
> > David Miller <davem@davemloft.net> wrote:
> >> It also means that the scope of developers who can contribute and work
> >> on the translater is much larger.
> >
> > How so? Translator is in userspace in nftables case too?
>
> Florian, first of all, the whole "change the iptables binary" idea is
> a non-starter. For the many reasons I have described in the various
> postings I have made today.
>
> It is entirely impractical.
???????
You suggest:
iptables -> setsockopt -> umh (xtables -> ebpf) -> kernel
How is this different from
iptables -> setsockopt -> umh (Xtables -> nftables -> kernel
?
EBPF can be placed within nftables either userspace or kernel,
there is nothing that prevents this.
> Anything designed in that nature must be distributed completely in the
> kernel tree, so that the iptables kernel ABI is provided without any
> externel dependencies.
Would you be willing to merge nftables into kernel tools directory then?
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:07 ` David Miller
@ 2018-02-19 15:20 ` Florian Westphal
2018-02-19 15:28 ` David Miller
0 siblings, 1 reply; 73+ messages in thread
From: Florian Westphal @ 2018-02-19 15:20 UTC (permalink / raw)
To: David Miller
Cc: fw, laforge, daniel, netdev, netfilter-devel, alexei.starovoitov
David Miller <davem@davemloft.net> wrote:
> From: Florian Westphal <fw@strlen.de>
> Date: Mon, 19 Feb 2018 15:53:14 +0100
>
> > Sure, but looking at all the things that were added to iptables
> > to alleviate some of the issues (ipset for instance) show that we need a
> > meaningful re-design of how things work conceptually.
>
> As you said iptables is in maintainenance mode.
>
> But there are millions upon millions of users, like it or not, and
> they aren't going away for decades. And this is the iptables binary
> ABI I'm talking about, not the iptables user command line interface.
I know.
> my house?" Please see further than the view inside your home.
>
> By in large, we are stuck with iptables's data path for an extremely
> long time.
So?
> Major data centers doesn't even enable NFTABLES in their kernels, and
> there is nothing you can do about that in the short to medium term.
So?
> Therefore, for all of the beneficial reasons I have discussed we
> should make that datapath as aligned and integrated with our core
> important technologies as possible, so that they can benefit from any
> and all improvements in that area rather than just collecting dust.
See my other mail, where I explained, in great detail, the problems
of the xtables UAPI.
If you go through with this, and, eventually somehow get feature parity,
all of the problems remain in full effect.
You will also need to replicate the translation efforts that already
went into nftables. The translator wasn't yet a high priority as we
lacked some features but this can be changed now that nft is catching
up.
Userspace program expectation is for iptables to be like fib for
instance, i.e. you can add and remove without stomping on each others
feet. You are setting this in stone.
You're also adding a way to make it so that I can delete entries from
the fib (bpfilter) but iproute2 will still show all entries (iptables
legacy).
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 14:44 ` David Miller
2018-02-19 14:53 ` Florian Westphal
@ 2018-02-19 15:23 ` Harald Welte
2018-02-19 15:32 ` David Miller
2018-02-19 15:36 ` David Miller
1 sibling, 2 replies; 73+ messages in thread
From: Harald Welte @ 2018-02-19 15:23 UTC (permalink / raw)
To: David Miller; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 09:44:51AM -0500, David Miller wrote:
> I see talk about "just replacing the iptables binary".
>
> A long time ago in a galaxy far far away, that would be a reasonable
> scheme. But that kind of approach won't work in the realities of
> today.
>
> You aren't going to be able to replace the iptables binary in the tens
> of thousands of container images out there, nor the virtualization
> installations either.
I appear to have been under the impression that the entire movement to
DevOps and automatic provisioning of containers/nodes/pods/Vms with
puppet, ansible, Dockerfiles & Co is to be *more* agile in deployments,
rather than less.
If you cannot even rebuild your thousands of container images with
updated binaries, then what is this all worth? You need to be able to
update the iptables (or any other) binary in case there's an important
(security or otherwise) bug that needs fixing. I don't see this any
different.
Also, as long as legacy ip_tables/x_tables is still in the kernel, you
can still run your old userspace against that old implementation in the
kernel. Nobody forces you to use anything else [for another decade or
so]. Just if you want to take advantage of new more
modular/performant/... things like nftables or an eBPF backend, then you
would have to go that extra mile.
I don't think the kernel (network) developers should burden themselves
with too many things. There's sufficient on their plate as-is. So
* if there's some new system (nftables, bpfilter, ...)
* and some documented migration paths for the vast majority of the use
cases (replacing iptables binaries with a compat wrapper)
* and the old system continues to work as-is (x_tables kernel code stays for
several more years)
Then people who care about the new features or performance will migrate
to the new system. And those who don't care stay with the old system -
which is not a problem as they clearly wouldn't need the new system
anyway.
> Like it or not iptables ABI based filtering is going to be in the data
> path for many years if not a decade or more to come.
I beg to differ. For some people, yes. but then, as Florian points
out, they can just as well use the existing x_tables kernel code. If
they want something better, they can either replace their iptables
program with xtables-compat from nftables, or whatever else might
exist for eBPF support.
> iptables is a victim of it's own success, like it or not :-) Yes, the
> ABI is terrible, but obviously it was useful enough for lots of
> people.
and it continues to do so. I just don't think it is a great idea
to kludge any new packet filter against such an arcane uapi.
> Therefore it behooves us to accept this reality and align the data
> path generated to match what the rest of the kernel is moving towards
> and that is eBPF and XDP.
This argument is unrelated to the question of the uapi. I'm not arguing
against an eBPF backend/implementation for packet filtering. It's more
a question of _how_.
> Furthrmore, on a long term maintainence perspective, it means that
> every data path used by the kernel for iptables will be fully verified
> by the eBPF verifier. This means that the iptables data path will be
> guaranteed to never get into a loop, access out of bounds data, etc.
>
> That to me is real power, and something we should pursue.
Once again, both not related to the question of the uapi.
> I know you can't see how offloading is possible, but I hope
> are some further discussion you can see how that might work.
I'm looking forward to that point.
Regards,
Harald
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:15 ` Florian Westphal
@ 2018-02-19 15:27 ` David Miller
2018-02-19 15:38 ` Harald Welte
2018-02-19 17:41 ` Arturo Borrero Gonzalez
2018-02-19 21:30 ` Jozsef Kadlecsik
1 sibling, 2 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 15:27 UTC (permalink / raw)
To: fw; +Cc: daniel, laforge, netdev, netfilter-devel, alexei.starovoitov
From: Florian Westphal <fw@strlen.de>
Date: Mon, 19 Feb 2018 16:15:55 +0100
> Would you be willing to merge nftables into kernel tools directory
> then?
Did you miss the part where I explained that people explicitly disable
NFTABLES in their kernel configs in most if not all large datacenters?
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:13 ` David Miller
2018-02-19 15:15 ` Florian Westphal
@ 2018-02-19 15:27 ` Harald Welte
2018-02-19 15:31 ` David Miller
1 sibling, 1 reply; 73+ messages in thread
From: Harald Welte @ 2018-02-19 15:27 UTC (permalink / raw)
To: David Miller; +Cc: fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 10:13:35AM -0500, David Miller wrote:
> Florian, first of all, the whole "change the iptables binary" idea is
> a non-starter. For the many reasons I have described in the various
> postings I have made today.
>
> It is entirely impractical.
Why is it practical to replace your kernel but not practical to replace
a small userspace tool running on top of it?
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:20 ` Florian Westphal
@ 2018-02-19 15:28 ` David Miller
0 siblings, 0 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 15:28 UTC (permalink / raw)
To: fw; +Cc: laforge, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Florian Westphal <fw@strlen.de>
Date: Mon, 19 Feb 2018 16:20:23 +0100
> See my other mail, where I explained, in great detail, the problems
> of the xtables UAPI.
As the person who wrote the bpfilter UAPI parser for this, you don't
need to explain this to me.
But it's not going anywhere, and is used by millions upon millions of
users.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:27 ` Harald Welte
@ 2018-02-19 15:31 ` David Miller
2018-02-19 17:09 ` Phil Sutter
` (2 more replies)
0 siblings, 3 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 15:31 UTC (permalink / raw)
To: laforge; +Cc: fw, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Harald Welte <laforge@gnumonks.org>
Date: Mon, 19 Feb 2018 16:27:46 +0100
> On Mon, Feb 19, 2018 at 10:13:35AM -0500, David Miller wrote:
>
>> Florian, first of all, the whole "change the iptables binary" idea is
>> a non-starter. For the many reasons I have described in the various
>> postings I have made today.
>>
>> It is entirely impractical.
>
> Why is it practical to replace your kernel but not practical to replace
> a small userspace tool running on top of it?
The container is just userspace components. Those are really baked in
and are never changing.
The hosting element, on the other hand, can upgrade the kernel in that
scenerio no problem.
This is how cloud hosting environments work.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:23 ` Harald Welte
@ 2018-02-19 15:32 ` David Miller
2018-02-19 15:37 ` Jan Engelhardt
2018-02-19 15:36 ` David Miller
1 sibling, 1 reply; 73+ messages in thread
From: David Miller @ 2018-02-19 15:32 UTC (permalink / raw)
To: laforge; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
From: Harald Welte <laforge@gnumonks.org>
Date: Mon, 19 Feb 2018 16:23:21 +0100
> Also, as long as legacy ip_tables/x_tables is still in the kernel, you
> can still run your old userspace against that old implementation in the
> kernel.
But without offloading, and the various other benefits which I have
tried to clearly explain to both you and Florian.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:23 ` Harald Welte
2018-02-19 15:32 ` David Miller
@ 2018-02-19 15:36 ` David Miller
2018-02-19 17:20 ` Harald Welte
` (2 more replies)
1 sibling, 3 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 15:36 UTC (permalink / raw)
To: laforge; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
From: Harald Welte <laforge@gnumonks.org>
Date: Mon, 19 Feb 2018 16:23:21 +0100
>> Like it or not iptables ABI based filtering is going to be in the data
>> path for many years if not a decade or more to come.
>
> I beg to differ. For some people, yes. but then, as Florian points
> out, they can just as well use the existing x_tables kernel code. If
> they want something better, they can either replace their iptables
> program with xtables-compat from nftables, or whatever else might
> exist for eBPF support.
nftables has been proported as "better" for years, yet large
institutions did not migrate to it. In fact, they explicitly
disabled NFTABLES in their kernel config.
You may want to ponder for a little while why that might be.
I think netfilter is at a real crossroads right now.
In my opinion, any resistence to integration with eBPF and XDP will
lead to even less adoption of netfilter as a technology.
Therefore my plan is to move everything to be integrated around these
important core technologies. For the purposes of integration, code
coverage, performance, and the ability to juxtapose different bits of
eBPF code into larger optimized code streams that can also be
offloaded into hardware.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:32 ` David Miller
@ 2018-02-19 15:37 ` Jan Engelhardt
2018-02-19 15:43 ` David Miller
0 siblings, 1 reply; 73+ messages in thread
From: Jan Engelhardt @ 2018-02-19 15:37 UTC (permalink / raw)
To: David Miller; +Cc: laforge, daniel, netdev, netfilter-devel, alexei.starovoitov
On Monday 2018-02-19 16:32, David Miller wrote:
>From: Harald Welte <laforge@gnumonks.org>
>Date: Mon, 19 Feb 2018 16:23:21 +0100
>
>> Also, as long as legacy ip_tables/x_tables is still in the kernel, you
>> can still run your old userspace against that old implementation in the
>> kernel.
>
>But without offloading, and the various other benefits which I have
>tried to clearly explain to both you and Florian.
Which is actually the business model to get people *off* the old ABI in
reasonable time. Otherwise, we would have to ask ourselves why we have
not yet enhanced /dev/raw with mmap and whatnot.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:27 ` David Miller
@ 2018-02-19 15:38 ` Harald Welte
2018-02-19 15:44 ` David Miller
2018-02-19 17:41 ` Arturo Borrero Gonzalez
1 sibling, 1 reply; 73+ messages in thread
From: Harald Welte @ 2018-02-19 15:38 UTC (permalink / raw)
To: David Miller; +Cc: fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Dear David,
On Mon, Feb 19, 2018 at 10:27:27AM -0500, David Miller wrote:
> > Would you be willing to merge nftables into kernel tools directory
> > then?
>
> Did you miss the part where I explained that people explicitly disable
> NFTABLES in their kernel configs in most if not all large datacenters?
If people to chose to disable a certain feature, then that is their own
decision to do so. We should respect that decision. Clearly they seem
to have no interest in a better or more featureful packet filter, then.
I mean, it's not like somebody proposes to implement NTFS inside the FAT
filesystem kernel module because distributors (or data centers) tend to
disable the NTFS module?!
How is kernel development these days constrained by what some users may
or may not put in their Kconfig? If they want a given feature, they
must enable it.
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:37 ` Jan Engelhardt
@ 2018-02-19 15:43 ` David Miller
0 siblings, 0 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 15:43 UTC (permalink / raw)
To: jengelh; +Cc: laforge, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Jan Engelhardt <jengelh@inai.de>
Date: Mon, 19 Feb 2018 16:37:57 +0100 (CET)
> On Monday 2018-02-19 16:32, David Miller wrote:
>
>>From: Harald Welte <laforge@gnumonks.org>
>>Date: Mon, 19 Feb 2018 16:23:21 +0100
>>
>>> Also, as long as legacy ip_tables/x_tables is still in the kernel, you
>>> can still run your old userspace against that old implementation in the
>>> kernel.
>>
>>But without offloading, and the various other benefits which I have
>>tried to clearly explain to both you and Florian.
>
> Which is actually the business model to get people *off* the old ABI in
> reasonable time.
Hosting companies can't change what customers run in their containers.
But if they are told that a kernel upgrade will get them offloading
and increase their performance termendously, then that gives them real
value.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:38 ` Harald Welte
@ 2018-02-19 15:44 ` David Miller
2018-02-19 17:14 ` Phil Sutter
0 siblings, 1 reply; 73+ messages in thread
From: David Miller @ 2018-02-19 15:44 UTC (permalink / raw)
To: laforge; +Cc: fw, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Harald Welte <laforge@gnumonks.org>
Date: Mon, 19 Feb 2018 16:38:08 +0100
> On Mon, Feb 19, 2018 at 10:27:27AM -0500, David Miller wrote:
>> > Would you be willing to merge nftables into kernel tools directory
>> > then?
>>
>> Did you miss the part where I explained that people explicitly disable
>> NFTABLES in their kernel configs in most if not all large datacenters?
>
> If people to chose to disable a certain feature, then that is their own
> decision to do so. We should respect that decision. Clearly they seem
> to have no interest in a better or more featureful packet filter, then.
>
> I mean, it's not like somebody proposes to implement NTFS inside the FAT
> filesystem kernel module because distributors (or data centers) tend to
> disable the NTFS module?!
>
> How is kernel development these days constrained by what some users may
> or may not put in their Kconfig? If they want a given feature, they
> must enable it.
This discussion was about why iptables UABI still matters.
And I'm trying to explain to you one of several reasons why it does.
Also, instead of saying "They decided to not use NFTABLES, oh well
that is their problem" it might be more beneficial, especially in the
long term for netfilter, to think about "why".
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:31 ` David Miller
@ 2018-02-19 17:09 ` Phil Sutter
2018-02-19 17:15 ` David Miller
2018-02-20 9:35 ` Michal Kubecek
2018-02-19 17:32 ` Harald Welte
2018-02-19 17:41 ` Arturo Borrero Gonzalez
2 siblings, 2 replies; 73+ messages in thread
From: Phil Sutter @ 2018-02-19 17:09 UTC (permalink / raw)
To: David Miller
Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 10:31:39AM -0500, David Miller wrote:
> From: Harald Welte <laforge@gnumonks.org>
> Date: Mon, 19 Feb 2018 16:27:46 +0100
>
> > On Mon, Feb 19, 2018 at 10:13:35AM -0500, David Miller wrote:
> >
> >> Florian, first of all, the whole "change the iptables binary" idea is
> >> a non-starter. For the many reasons I have described in the various
> >> postings I have made today.
> >>
> >> It is entirely impractical.
> >
> > Why is it practical to replace your kernel but not practical to replace
> > a small userspace tool running on top of it?
>
> The container is just userspace components. Those are really baked in
> and are never changing.
Which is a problem per se. Cheap hardware routers are a good example of
why business models which tend to get customers stuck with old software
have such dramatic effects at least in matters of security.
> The hosting element, on the other hand, can upgrade the kernel in that
> scenerio no problem.
>
> This is how cloud hosting environments work.
What puzzles me about your argumentation is that you seem to propose for
the kernel to cover up flaws in userspace. Spinning this concept further
would mean that if there would be an old bug in iproute2 we should think
of adding a workaround to rtnetlink interface in kernel because
containers will keep the old iproute2 binary? Or am I (hopefully) just
missing your point?
Cheers, Phil
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:44 ` David Miller
@ 2018-02-19 17:14 ` Phil Sutter
2018-02-19 17:22 ` David Miller
0 siblings, 1 reply; 73+ messages in thread
From: Phil Sutter @ 2018-02-19 17:14 UTC (permalink / raw)
To: David Miller
Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 10:44:59AM -0500, David Miller wrote:
> From: Harald Welte <laforge@gnumonks.org>
> Date: Mon, 19 Feb 2018 16:38:08 +0100
>
> > On Mon, Feb 19, 2018 at 10:27:27AM -0500, David Miller wrote:
> >> > Would you be willing to merge nftables into kernel tools directory
> >> > then?
> >>
> >> Did you miss the part where I explained that people explicitly disable
> >> NFTABLES in their kernel configs in most if not all large datacenters?
> >
> > If people to chose to disable a certain feature, then that is their own
> > decision to do so. We should respect that decision. Clearly they seem
> > to have no interest in a better or more featureful packet filter, then.
> >
> > I mean, it's not like somebody proposes to implement NTFS inside the FAT
> > filesystem kernel module because distributors (or data centers) tend to
> > disable the NTFS module?!
> >
> > How is kernel development these days constrained by what some users may
> > or may not put in their Kconfig? If they want a given feature, they
> > must enable it.
>
> This discussion was about why iptables UABI still matters.
>
> And I'm trying to explain to you one of several reasons why it does.
>
> Also, instead of saying "They decided to not use NFTABLES, oh well
> that is their problem" it might be more beneficial, especially in the
> long term for netfilter, to think about "why".
OK, so reading between the lines you're saying that nftables project has
failed to provide an adequate successor to iptables?
Cheers, Phil
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:09 ` Phil Sutter
@ 2018-02-19 17:15 ` David Miller
2018-02-20 13:05 ` Phil Sutter
2018-02-20 9:35 ` Michal Kubecek
1 sibling, 1 reply; 73+ messages in thread
From: David Miller @ 2018-02-19 17:15 UTC (permalink / raw)
To: phil; +Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Phil Sutter <phil@nwl.cc>
Date: Mon, 19 Feb 2018 18:09:39 +0100
> What puzzles me about your argumentation is that you seem to propose for
> the kernel to cover up flaws in userspace. Spinning this concept further
> would mean that if there would be an old bug in iproute2 we should think
> of adding a workaround to rtnetlink interface in kernel because
> containers will keep the old iproute2 binary? Or am I (hopefully) just
> missing your point?
I'll answer this with a question. I tried to remove UFO entirely from
the kernel, did you see how that went?
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:36 ` David Miller
@ 2018-02-19 17:20 ` Harald Welte
2018-02-19 17:29 ` David Miller
2018-02-19 17:40 ` Arturo Borrero Gonzalez
2018-02-19 18:06 ` Arturo Borrero Gonzalez
2 siblings, 1 reply; 73+ messages in thread
From: Harald Welte @ 2018-02-19 17:20 UTC (permalink / raw)
To: David Miller; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 10:36:51AM -0500, David Miller wrote:
> nftables has been proported as "better" for years, yet large
> institutions did not migrate to it. In fact, they explicitly
> disabled NFTABLES in their kernel config.
It's like with any migration. People were using ipchains for a long
time even after iptables existed. Many people simply don't care
about packet filter performance. It's only a small fraction of their
entire CPU workload, so probably not worth optimzing. For dedicated
firewall devices, that's of course a different story.
How long did it take for the getrandom() system call to be actually used
by applications [even glibc!]? Or many other things that get introduced
in the kernel?
I can just as well ask how many millions of users / devices are already
using eBPF or XDP? How many major Linux distributions are enabling
and/or supporting this yet? I'm not criticizing, I'm just attempting
to illustrate that technologies always take time to establish
themselves - and of course those people with the biggest benefit (and
knowing about it) will be the early adopters, while many others have no
motivation to migrate.
> In my opinion, any resistence to integration with eBPF and XDP will
> lead to even less adoption of netfilter as a technology.
1) I may not have made my point clear, sorry. I have not argued against
any integration with eBPF, I have just made some specific arguments
against specific aspects of the current RFC.
2) You have indicated repeatedly that there are millions and millions of
netfilter/iptables users out there. So I fail to see the "even less
adoption" part. "Even less" than those millions and millions? SCNR.
Regards,
Harald
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:14 ` Phil Sutter
@ 2018-02-19 17:22 ` David Miller
2018-02-19 18:05 ` Phil Sutter
` (2 more replies)
0 siblings, 3 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 17:22 UTC (permalink / raw)
To: phil; +Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Phil Sutter <phil@nwl.cc>
Date: Mon, 19 Feb 2018 18:14:11 +0100
> OK, so reading between the lines you're saying that nftables project
> has failed to provide an adequate successor to iptables?
Whilst it is great that the atomic table update problem was solved, I
think the emphasis on flexibility often at the expense of performance
was a bad move.
Netfilter's chronic performance differential is why a lot of mindshare
was lost to userspace networking technologies.
Thankfully, we are gaining back a lot of that userbase with XDP and
eBPF, thanks to the hard work of many individuals.
To think that people are going to be willing to take the performance
hit (whatever it's size) to go back to the "more flexible" nftables
is really not a realistic expectation.
And we have amassed enough interest and momentum that offloading eBPF
in hardware on current and future hardware is happening.
So I am going to direct us in directions that allow those realities to
be taken advantage of, rather than pretending that this transition
hasn't occurred already.
Thank you.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:20 ` Harald Welte
@ 2018-02-19 17:29 ` David Miller
2018-02-19 18:37 ` Harald Welte
0 siblings, 1 reply; 73+ messages in thread
From: David Miller @ 2018-02-19 17:29 UTC (permalink / raw)
To: laforge; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
From: Harald Welte <laforge@gnumonks.org>
Date: Mon, 19 Feb 2018 18:20:40 +0100
> It's like with any migration. People were using ipchains for a long
> time even after iptables existed. Many people simply don't care
> about packet filter performance. It's only a small fraction of their
> entire CPU workload, so probably not worth optimzing. For dedicated
> firewall devices, that's of course a different story.
"I have power in my house, what's the big deal about this power
outage I hear about?"
People with an Android phone in their pocket is using iptables, and
the overhead and performance of those rules really does matter. It
determines how long your battery life is, etc.
> I can just as well ask how many millions of users / devices are
> already using eBPF or XDP?
Every time someone connects to a major provider, they are using it.
And by in large, for system tracing and analysis eBPF is basically
a hard requirement for people doing anything serious these days.
Please see the wonderful work by Brendan Gregg and others which has
basically made the GPL'ing of DTrace by Oracle entirely irrelevant and
our Linux's tracing infrastructure has become must more powerful and
capable thanks to eBPF.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:31 ` David Miller
2018-02-19 17:09 ` Phil Sutter
@ 2018-02-19 17:32 ` Harald Welte
2018-02-19 17:41 ` Arturo Borrero Gonzalez
2 siblings, 0 replies; 73+ messages in thread
From: Harald Welte @ 2018-02-19 17:32 UTC (permalink / raw)
To: David Miller; +Cc: fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 10:31:39AM -0500, David Miller wrote:
> > Why is it practical to replace your kernel but not practical to replace
> > a small userspace tool running on top of it?
>
> The container is just userspace components. Those are really baked in
> and are never changing.
never until you have to apply a bug fix for any of the many components you bake
into it. I am doing this on an (at least) weekly basis for my Docker containers.
That's no different from a classic Linux distribution where you update your apt/rpm
packages all the time.
A container that is static and cannot continuously updated with new versions
for security (and other) fixes is broken by design. If some people are doing
this, they IMHO have no sense of IT security, and such usage pattersn are not
what kernel development should cite as primary use case (again IMHO).
> This is how cloud hosting environments work.
Yes, *one* particular use case. By far not every use case of Linux, or
Linux packet filtering.
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:36 ` David Miller
2018-02-19 17:20 ` Harald Welte
@ 2018-02-19 17:40 ` Arturo Borrero Gonzalez
2018-02-19 18:06 ` Arturo Borrero Gonzalez
2 siblings, 0 replies; 73+ messages in thread
From: Arturo Borrero Gonzalez @ 2018-02-19 17:40 UTC (permalink / raw)
To: David Miller
Cc: laforge, daniel, netdev, Netfilter Development Mailing list,
alexei.starovoitov
On 19 February 2018 at 16:36, David Miller <davem@davemloft.net> wrote:
>
> I think netfilter is at a real crossroads right now.
>
I don't think so. The Netfilter Project and the Netfilter Community
already "agreed" on nftables and we are working on it.
But this isn't a secret, right? We have been open-discussing and
open-working on this for *years* now.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:31 ` David Miller
2018-02-19 17:09 ` Phil Sutter
2018-02-19 17:32 ` Harald Welte
@ 2018-02-19 17:41 ` Arturo Borrero Gonzalez
2 siblings, 0 replies; 73+ messages in thread
From: Arturo Borrero Gonzalez @ 2018-02-19 17:41 UTC (permalink / raw)
To: David Miller
Cc: laforge, Florian Westphal, daniel, netdev,
Netfilter Development Mailing list, alexei.starovoitov
[-- Attachment #1: Type: text/plain, Size: 490 bytes --]
On 19 February 2018 at 16:31, David Miller <davem@davemloft.net> wrote:
>
> This is how cloud hosting environments work.
BTW, to give a bit more context since it has been mentioned several
times already, find attached 2 rulesets from real-life cloud
deployments: openstack and kubernetes.
These rules were generated for iptables by means of either iptables or
iptables-save.
Both of these rulesets can be easily (automagically) translated to
nftables with code which is already released.
[-- Attachment #2: k8s-iptables.txt --]
[-- Type: text/plain, Size: 184166 bytes --]
# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*nat
:PREROUTING ACCEPT XX:XX
:INPUT ACCEPT XX:XX
:OUTPUT ACCEPT XX:XXXX
:POSTROUTING ACCEPT XX:XXXX
:FLANNEL - XX:XX
:KUBE-MARK-DROP - XX:XX
:KUBE-MARK-MASQ - XX:XX
:KUBE-NODEPORTS - XX:XX
:KUBE-POSTROUTING - XX:XX
:KUBE-SEP-XXDIJNCXZNXRXCDK - XX:XX
:KUBE-SEP-XXNKLXKTWBXXGGRM - XX:XX
:KUBE-SEP-XXOOEPTHROQWXXZB - XX:XX
:KUBE-SEP-XGCOHXVPDXTXIZUS - XX:XX
:KUBE-SEP-XIIVCPFXKMCXHWBX - XX:XX
:KUBE-SEP-XMIJSFZXSOXERHCE - XX:XX
:KUBE-SEP-XTJZYLXVVPCOXLWH - XX:XX
:KUBE-SEP-XXJNSHVLAXXBZXYX - XX:XX
:KUBE-SEP-XGWLAZBOXUKXFXKX - XX:XX
:KUBE-SEP-XIAXOXIVJMCHKMXX - XX:XX
:KUBE-SEP-XKNYFWSSIXNVVCPB - XX:XX
:KUBE-SEP-XLTTNSWMTXQCZQTJ - XX:XX
:KUBE-SEP-XOXOKVYPYXKTUSSB - XX:XX
:KUBE-SEP-XVREBNCXQXVXUHUW - XX:XX
:KUBE-SEP-XXRXZAXCZYXDZMMA - XX:XX
:KUBE-SEP-XYSFKYVNDXOUESZX - XX:XX
:KUBE-SEP-XXCSXBXVBPXHXHVE - XX:XX
:KUBE-SEP-XXLXYXSXXXYXQXYX - XX:XX
:KUBE-SEP-XXBNXAGOXXXFKJYX - XX:XX
:KUBE-SEP-XXDXHIGQIZHPOBEL - XX:XX
:KUBE-SEP-XXZBGBXOJKUEVCXO - XX:XX
:KUBE-SEP-XXIXWXTXXZWQYCUF - XX:XX
:KUBE-SEP-XAXMWFMJJXXXGWJU - XX:XX
:KUBE-SEP-XASJXLVYYABDOXRR - XX:XX
:KUBE-SEP-XBFJMUPXHAZLXXXE - XX:XX
:KUBE-SEP-XBGXPXYXLKZVXXIA - XX:XX
:KUBE-SEP-XHRAZUVCRXGRYEXX - XX:XX
:KUBE-SEP-XIAZXNFOJXDGWPXX - XX:XX
:KUBE-SEP-XINDJEXBGAFNKMDK - XX:XX
:KUBE-SEP-XIXCIFAXDJVZTSRL - XX:XX
:KUBE-SEP-XRCMSDHCNWFSGCZG - XX:XX
:KUBE-SEP-XRHHCDXTPWEGIXLX - XX:XX
:KUBE-SEP-XRZGCKXKVSUWTXVX - XX:XX
:KUBE-SEP-XVRXIRHFHURMXKXB - XX:XX
:KUBE-SEP-XXODXIDXOHNUJZWZ - XX:XX
:KUBE-SEP-XXPJZHXOJGZJFDYA - XX:XX
:KUBE-SEP-XAHRYQUUXYXIDXSI - XX:XX
:KUBE-SEP-XGXYXOQECSCHYQSM - XX:XX
:KUBE-SEP-XIQWDXXBCXVOXZXB - XX:XX
:KUBE-SEP-XKNFNNDXSFQKMGDR - XX:XX
:KUBE-SEP-XMOMWXXUCGJWUDWX - XX:XX
:KUBE-SEP-XNAAXXOPTCMXKXYM - XX:XX
:KUBE-SEP-XNFXZXYXWDJZKDLD - XX:XX
:KUBE-SEP-XRPQZGHSFAJEXXIP - XX:XX
:KUBE-SEP-XWPBVGXXTDFXFYDG - XX:XX
:KUBE-SEP-XXLXWSLUXFXTXSXT - XX:XX
:KUBE-SEP-XXKCVBUXXXLZXVQG - XX:XX
:KUBE-SEP-XXXXILOCXGNBWYKX - XX:XX
:KUBE-SEP-XBGYCPBSKXXKXOOJ - XX:XX
:KUBE-SEP-XBIXEZSSCVBXOTPG - XX:XX
:KUBE-SEP-XCFAFXXPXUXXNXXX - XX:XX
:KUBE-SEP-XGTIANXWGTJQCXNE - XX:XX
:KUBE-SEP-XHFUTFXXXOSXVVHH - XX:XX
:KUBE-SEP-XIXRRGOIWSTKXYSX - XX:XX
:KUBE-SEP-XSJLKSXTXXCQXCXX - XX:XX
:KUBE-SEP-XZXXWIXGPGHXFHSX - XX:XX
:KUBE-SEP-XXIONLXMLGFZOZFB - XX:XX
:KUBE-SEP-XXPIAXSAIFUHQGND - XX:XX
:KUBE-SEP-XAXPXGVMAZEXPTBX - XX:XX
:KUBE-SEP-XBKXXGQQCWXLAXYD - XX:XX
:KUBE-SEP-XCJATFMBWNQXFXNS - XX:XX
:KUBE-SEP-XGQXNUXUVVREFJGE - XX:XX
:KUBE-SEP-XHXSXRXVRXXXFOUP - XX:XX
:KUBE-SEP-XHJPKSWXXNYXZXXC - XX:XX
:KUBE-SEP-XPNHQHILSTXODFPD - XX:XX
:KUBE-SEP-XPUCLUJXGNJMQURT - XX:XX
:KUBE-SEP-XSXIXXXPHIHIPXLE - XX:XX
:KUBE-SEP-XUCFXAHUXAXXXALA - XX:XX
:KUBE-SEP-XUJHIJKZBXHGAKXS - XX:XX
:KUBE-SEP-XZXRBDLLPAKXIPZD - XX:XX
:KUBE-SEP-AEXDRVALKCXYXGMW - XX:XX
:KUBE-SEP-AEVCABFEIEGRXTEK - XX:XX
:KUBE-SEP-AHAFXLXAGUXXQUXP - XX:XX
:KUBE-SEP-AJXXRTXONUXGZJXT - XX:XX
:KUBE-SEP-AJFMIXBSJXVEXXXT - XX:XX
:KUBE-SEP-AKXTJHXIXXXYKOXE - XX:XX
:KUBE-SEP-AKQXXGVYESGFXLFO - XX:XX
:KUBE-SEP-ALFFLJPXKDVXNTNX - XX:XX
:KUBE-SEP-AMGCZYBIAHHUXXXX - XX:XX
:KUBE-SEP-AMJEXFSWNSJLDDAC - XX:XX
:KUBE-SEP-APXHMALBXDXHLZCX - XX:XX
:KUBE-SEP-AROFZVLFCAODIYUK - XX:XX
:KUBE-SEP-ATXKFPRQXCTRNXSE - XX:XX
:KUBE-SEP-BXXBQRXTLFXNXDJX - XX:XX
:KUBE-SEP-BHRJDNOQXXZKRTUC - XX:XX
:KUBE-SEP-BLMPNFXAIDYJSKXO - XX:XX
:KUBE-SEP-BRDXGGFXXYBFPSJT - XX:XX
:KUBE-SEP-BYGNXIVTCZNXXXNV - XX:XX
:KUBE-SEP-CXXZIBLLCLRNKDXX - XX:XX
:KUBE-SEP-CBXTTPIDEXKXXNEV - XX:XX
:KUBE-SEP-CDBMXXGNMILHQLXV - XX:XX
:KUBE-SEP-CEOAQHQEICXMXXHD - XX:XX
:KUBE-SEP-CIQZXXPZXWDZXOAK - XX:XX
:KUBE-SEP-CPCDSSDEDRQBVJCV - XX:XX
:KUBE-SEP-CXXLQHOXTXTRPOVG - XX:XX
:KUBE-SEP-CXXXSQXWXXXJBIYX - XX:XX
:KUBE-SEP-CZVAWNRXDWMSIVNA - XX:XX
:KUBE-SEP-DXJVGTXXONKLONXZ - XX:XX
:KUBE-SEP-DXIZHYZXQEXBXXYQ - XX:XX
:KUBE-SEP-DBMRXUMTBXVRSJXD - XX:XX
:KUBE-SEP-DKXMXFNXTBESXDLE - XX:XX
:KUBE-SEP-DMNYWOPVWRDKXOTX - XX:XX
:KUBE-SEP-DNMECZZXLFQXXIMD - XX:XX
:KUBE-SEP-DUUXNYOZROGKKSIM - XX:XX
:KUBE-SEP-DXXFXHFXXLXZOBGD - XX:XX
:KUBE-SEP-EXFOQXDFXSZXZIDW - XX:XX
:KUBE-SEP-EXXNBXDNWCWGTXQT - XX:XX
:KUBE-SEP-EDASXVXXMMUKUEXX - XX:XX
:KUBE-SEP-EMSIJAKTGRDEXXGJ - XX:XX
:KUBE-SEP-EQOLXXXHXOXKXSHX - XX:XX
:KUBE-SEP-ERXMCXSJXRQRXOXE - XX:XX
:KUBE-SEP-ESSREXJUXFEYEUTH - XX:XX
:KUBE-SEP-ESSSFRXQUDXIXXQC - XX:XX
:KUBE-SEP-ESYUEXZIXMXXOZUX - XX:XX
:KUBE-SEP-EVXXCFGMAOUPYJRU - XX:XX
:KUBE-SEP-EWKXSTBCSZXXHBYH - XX:XX
:KUBE-SEP-EXDXURVOXUWSRDXD - XX:XX
:KUBE-SEP-FXXVXQJVXXXZMMEX - XX:XX
:KUBE-SEP-FCNMQLICDXZXHGPS - XX:XX
:KUBE-SEP-FDFXZYXLXZOXUNRB - XX:XX
:KUBE-SEP-FEDZSXKERFEDKMDY - XX:XX
:KUBE-SEP-FFLSSXXXCURPOXJJ - XX:XX
:KUBE-SEP-FMBHPSBNXPJHFQMN - XX:XX
:KUBE-SEP-FNQXHMYFXFGLPUXC - XX:XX
:KUBE-SEP-FOCKHSOXXYZXGXMQ - XX:XX
:KUBE-SEP-FQXVOLQGTXXBXUXN - XX:XX
:KUBE-SEP-FTDXXXXXGPTBZRVX - XX:XX
:KUBE-SEP-FTSYCICMKRQYJBZH - XX:XX
:KUBE-SEP-FWTRXOFKSAINOBXB - XX:XX
:KUBE-SEP-GXXRIGDZOHBSSXPX - XX:XX
:KUBE-SEP-GXXZXMQXOXAOJXBS - XX:XX
:KUBE-SEP-GACXVHRXIXXNCYSG - XX:XX
:KUBE-SEP-GAOPRXWQBCCEKBON - XX:XX
:KUBE-SEP-GCUUDEUAQBSJXNNX - XX:XX
:KUBE-SEP-GHGUXUSZFMXIMXIB - XX:XX
:KUBE-SEP-GIDXXMFAPLTJTTVV - XX:XX
:KUBE-SEP-GKAYASHWKXGHUSFX - XX:XX
:KUBE-SEP-GPUXOXSQSFYNYWXX - XX:XX
:KUBE-SEP-GQXKJUBTXXZXSPQV - XX:XX
:KUBE-SEP-GXXGWXLLFCXLDXDF - XX:XX
:KUBE-SEP-HXJCXHXXOTYFZRXX - XX:XX
:KUBE-SEP-HXXRYXGSAACJKMQD - XX:XX
:KUBE-SEP-HATXEDGCLVXINEXH - XX:XX
:KUBE-SEP-HGIVSFRTXXAXNHXX - XX:XX
:KUBE-SEP-HSFDSOYCXLBMRHXT - XX:XX
:KUBE-SEP-HYEKGUMHLPORRXFX - XX:XX
:KUBE-SEP-HYQKNCVLILQXQYQY - XX:XX
:KUBE-SEP-IXVXPLSXZBBVDACE - XX:XX
:KUBE-SEP-IXGKMHTPNPCKXDQF - XX:XX
:KUBE-SEP-IDXXIXVVUPCWOZBX - XX:XX
:KUBE-SEP-IFXCFXFXYZSDGXDC - XX:XX
:KUBE-SEP-IMXOZNUHPVKNJOCG - XX:XX
:KUBE-SEP-IOXBCMXXSLXSAJXO - XX:XX
:KUBE-SEP-IUXCMFIVXXTXAINK - XX:XX
:KUBE-SEP-IXXLVXULXQXXYCKG - XX:XX
:KUBE-SEP-IZPCXCXQXRXMHVLX - XX:XX
:KUBE-SEP-JXSXTLBYGWDHBQXS - XX:XX
:KUBE-SEP-JXZYVTIQGTWNQGWA - XX:XX
:KUBE-SEP-JFEXFKXXXMXSDMVI - XX:XX
:KUBE-SEP-JOBNTVXNLXDAWAXX - XX:XX
:KUBE-SEP-JPXKXXNXYPXQTZXE - XX:XX
:KUBE-SEP-JRZSWCXXXXDAXQCH - XX:XX
:KUBE-SEP-JUJXSSMXDGDXUKCX - XX:XX
:KUBE-SEP-KXCBJXXTNTJXTEIX - XX:XX
:KUBE-SEP-KEXDLPXIMJFBYEQX - XX:XX
:KUBE-SEP-KKXXOIXXCXFYGMBH - XX:XX
:KUBE-SEP-KKJKRURKHXHTIRRW - XX:XX
:KUBE-SEP-KLXTPYGHNGBNNDXC - XX:XX
:KUBE-SEP-KRXXMFEWCAXXLXXF - XX:XX
:KUBE-SEP-KVZCXXXXQNMSXZZF - XX:XX
:KUBE-SEP-LXYLUAILFMGAZHGA - XX:XX
:KUBE-SEP-LACEFMXTWPWGIOTJ - XX:XX
:KUBE-SEP-LBOGUSDMBAJJDOXK - XX:XX
:KUBE-SEP-LHBORJVHQFXXDEHT - XX:XX
:KUBE-SEP-LHHXGNGPUKWXONLW - XX:XX
:KUBE-SEP-LKAXZUEIEXHGXEXX - XX:XX
:KUBE-SEP-LQAALXZAKRVXBRXA - XX:XX
:KUBE-SEP-LYZJOBIRJOXAWBOO - XX:XX
:KUBE-SEP-LZXJDAONXXXBPHXB - XX:XX
:KUBE-SEP-MXMIWXXDLEUVOERO - XX:XX
:KUBE-SEP-MXBXXCCFXMSUDXOB - XX:XX
:KUBE-SEP-MAXXHNMXYXTIXXXH - XX:XX
:KUBE-SEP-MMFRTYIXXTIXBJQU - XX:XX
:KUBE-SEP-MNCXKIDAGVJPJJKO - XX:XX
:KUBE-SEP-MOXDXQXIOXBELXWX - XX:XX
:KUBE-SEP-MQEWOJWPHPVOHDNX - XX:XX
:KUBE-SEP-MRXXFQXXXFCPRDEF - XX:XX
:KUBE-SEP-MXDEFXGZSWCWDDXM - XX:XX
:KUBE-SEP-NXXHEXHAJBWZANYE - XX:XX
:KUBE-SEP-NIXDSDLDDNYDRCQN - XX:XX
:KUBE-SEP-NLSXKXNQEAZULNJX - XX:XX
:KUBE-SEP-NMYESYJXHAMBXEVX - XX:XX
:KUBE-SEP-NUJRJIJXJSISCXUL - XX:XX
:KUBE-SEP-NXGXLAEZDXDQCTXJ - XX:XX
:KUBE-SEP-NZIXOBCPMZOIAZFA - XX:XX
:KUBE-SEP-OXRXPXXDOHCVUAEM - XX:XX
:KUBE-SEP-OXNXGYBXXNHVTWXG - XX:XX
:KUBE-SEP-OIYKXYEXNOXFYXDX - XX:XX
:KUBE-SEP-OLXUXSFXWZAQXQXJ - XX:XX
:KUBE-SEP-OPAVXQXNDINUXESF - XX:XX
:KUBE-SEP-OPJXLKWNWPQXVXXH - XX:XX
:KUBE-SEP-ORLFLCRNYFTCFDXT - XX:XX
:KUBE-SEP-OSKWUDBAZCYEXMXX - XX:XX
:KUBE-SEP-PXWUXNTCIGZXONUA - XX:XX
:KUBE-SEP-PXEIHHMPWXRIOTXZ - XX:XX
:KUBE-SEP-PBKBKAYRMXOXHFLQ - XX:XX
:KUBE-SEP-PDOBQZHDNANTXLZX - XX:XX
:KUBE-SEP-PFXQQXCBPXFCJOXS - XX:XX
:KUBE-SEP-PIEADXBQPJGHEXIW - XX:XX
:KUBE-SEP-PPZLJYXXUZXKKXSS - XX:XX
:KUBE-SEP-PQFZVDXLXXXCDDHZ - XX:XX
:KUBE-SEP-QXMKRQTRXXYGNDSU - XX:XX
:KUBE-SEP-QBJAQUBXHUBBUBXT - XX:XX
:KUBE-SEP-QECIAEFZDXHXIRNQ - XX:XX
:KUBE-SEP-QJZXXNXDTXZXGYWC - XX:XX
:KUBE-SEP-QOFMVTGKEPYMXNJX - XX:XX
:KUBE-SEP-QRXUQSXMSFDVZMYZ - XX:XX
:KUBE-SEP-RXZZWVXAXAIFMFXX - XX:XX
:KUBE-SEP-RXVOXXZRPXWXGXGL - XX:XX
:KUBE-SEP-RBXHXYAUHDGHENDL - XX:XX
:KUBE-SEP-REXPQNMGXXAASXPS - XX:XX
:KUBE-SEP-RRJHMMXBCKNFAXGU - XX:XX
:KUBE-SEP-RSXTXXXEJMXPXOXE - XX:XX
:KUBE-SEP-RTVFTXXXSJDZUTXX - XX:XX
:KUBE-SEP-SXQXAJMXYMXXXTEX - XX:XX
:KUBE-SEP-SLXJBTMZUNNFRHKS - XX:XX
:KUBE-SEP-STZMJAGTXUPYRKXT - XX:XX
:KUBE-SEP-SUADNDUYKXMQLALM - XX:XX
:KUBE-SEP-SVAQIAHXEXXVXNXP - XX:XX
:KUBE-SEP-SZXOYSXJXXQFXXIT - XX:XX
:KUBE-SEP-TXMKXKFHXZXXWIXK - XX:XX
:KUBE-SEP-TXHXCMUJUGEXIIVX - XX:XX
:KUBE-SEP-TXWRXXLDXUJMPSSU - XX:XX
:KUBE-SEP-TXXYJFUREXXDLLTG - XX:XX
:KUBE-SEP-TMXAZNXUSICXXWMB - XX:XX
:KUBE-SEP-TOQEXXXPOKTVCDEA - XX:XX
:KUBE-SEP-TPXFEZTUIKWENXLX - XX:XX
:KUBE-SEP-TVLLXEGQZHXXJBKW - XX:XX
:KUBE-SEP-UXXPEXMZJDJFNHXX - XX:XX
:KUBE-SEP-UBYTXZMXSEHXXWXC - XX:XX
:KUBE-SEP-UCXJXFLUUUMRJJYA - XX:XX
:KUBE-SEP-UCETASOSIXJDXZXH - XX:XX
:KUBE-SEP-UDXXXXXXTOXMPXZC - XX:XX
:KUBE-SEP-UMGLSKFVXXUSOMTL - XX:XX
:KUBE-SEP-UONYXWXXRHJALCSS - XX:XX
:KUBE-SEP-URUQSXXXPEXDMXTF - XX:XX
:KUBE-SEP-USVXTXXIKXLBNCPB - XX:XX
:KUBE-SEP-UTXOXFNFABWHVGCX - XX:XX
:KUBE-SEP-VIAHEXHEXXDPVFUX - XX:XX
:KUBE-SEP-VJXNTJXTXJRNWPUD - XX:XX
:KUBE-SEP-VKDULWHHBYJKYKZX - XX:XX
:KUBE-SEP-VKLMPMSXDXBEISCW - XX:XX
:KUBE-SEP-VKXPCXTMFURRPXXX - XX:XX
:KUBE-SEP-VNSXSXXRGGYEHXXY - XX:XX
:KUBE-SEP-VSPXRXBFDYRLZIHC - XX:XX
:KUBE-SEP-VTFOGXMYBZLEXXUT - XX:XX
:KUBE-SEP-VVNXTXOFJOXIVXXY - XX:XX
:KUBE-SEP-VXXXDZZJXXYXCAXV - XX:XX
:KUBE-SEP-VZSUSPSQYXXCXCXP - XX:XX
:KUBE-SEP-WNNUSXXCXBXESMXA - XX:XX
:KUBE-SEP-WWXKKXPXZISFXWYY - XX:XX
:KUBE-SEP-XXNGAFXPKVDCBEOE - XX:XX
:KUBE-SEP-XBIBKZIWRXQFUXIU - XX:XX
:KUBE-SEP-XCFYDDJVFZXSQJXU - XX:XX
:KUBE-SEP-XGHZLNYDTXGPXVXZ - XX:XX
:KUBE-SEP-XHRCRWHRSCXVLUUG - XX:XX
:KUBE-SEP-XJBHGVMWXXSXXJIA - XX:XX
:KUBE-SEP-XPXNRZRYBXXBWDMI - XX:XX
:KUBE-SEP-XPYSZLKAPUWWUNKD - XX:XX
:KUBE-SEP-XUNLVRURGXZOTLXJ - XX:XX
:KUBE-SEP-XWDLVELPFVUPQTXV - XX:XX
:KUBE-SEP-YXXSVELPXQSJXFEB - XX:XX
:KUBE-SEP-YXXUHBBVXRPKUPPY - XX:XX
:KUBE-SEP-YXEQXVIXWLOAYPXY - XX:XX
:KUBE-SEP-YBIJXQXYCNAAFMSX - XX:XX
:KUBE-SEP-YMVXFQWJTNYIXUQX - XX:XX
:KUBE-SEP-YPSRQIRBZPXSMDAW - XX:XX
:KUBE-SEP-YUXKYZOAIIOLXPGJ - XX:XX
:KUBE-SEP-ZXJGHRWWGEGRZTKX - XX:XX
:KUBE-SEP-ZAMXLSXVYXXBRUKU - XX:XX
:KUBE-SEP-ZBQPUGNYSXXHSKOO - XX:XX
:KUBE-SEP-ZEXMXXGXLXXAXQJF - XX:XX
:KUBE-SEP-ZFJYNOXNYDZYNUXH - XX:XX
:KUBE-SEP-ZHXAUOTEHBQMELPQ - XX:XX
:KUBE-SEP-ZHDHXBXJLGXDXRXD - XX:XX
:KUBE-SEP-ZHWIXXKLTIJEXKET - XX:XX
:KUBE-SEP-ZPXIDVIKXOVXXCIA - XX:XX
:KUBE-SEP-ZZXRXUIHXXXYXIRL - XX:XX
:KUBE-SERVICES - XX:XX
:KUBE-SVC-XXXUVXNQMWXYSWTK - XX:XX
:KUBE-SVC-XXNXPPPXLHXVSKXT - XX:XX
:KUBE-SVC-XXDXXMLSXXZZXWBX - XX:XX
:KUBE-SVC-XAMYINUXQIQJNTKI - XX:XX
:KUBE-SVC-XDALDXMRMYUXXXXW - XX:XX
:KUBE-SVC-XDDEXMFNHNSQHKLA - XX:XX
:KUBE-SVC-XFEADOKDMXXTPSOI - XX:XX
:KUBE-SVC-XIXXHOZKPQXBJDXF - XX:XX
:KUBE-SVC-XNWHOQJXZXYDXQAX - XX:XX
:KUBE-SVC-XPCRIRAXZQCLXXQO - XX:XX
:KUBE-SVC-XQSGXKGXCPLFWVQC - XX:XX
:KUBE-SVC-XUNDXXKWWLCXFMXB - XX:XX
:KUBE-SVC-XXWITZOXXXKQZSJN - XX:XX
:KUBE-SVC-XAXGXYZSMCBVNXJU - XX:XX
:KUBE-SVC-XAXFXWWXXMYXFKAX - XX:XX
:KUBE-SVC-XALVSUXXVMSRVJDD - XX:XX
:KUBE-SVC-XBJXXGHYXBSXSPHX - XX:XX
:KUBE-SVC-XCJBOXYMSYXMXDAT - XX:XX
:KUBE-SVC-XCMEVYXYGATYYYXX - XX:XX
:KUBE-SVC-XKAXXYRMXXPFXBND - XX:XX
:KUBE-SVC-XSPOKYTTXEMSQXOH - XX:XX
:KUBE-SVC-XVLUXXXUGKEGEXJC - XX:XX
:KUBE-SVC-XWJXIXXNJYJDGIXB - XX:XX
:KUBE-SVC-XXCEIMYAEEXBXXTZ - XX:XX
:KUBE-SVC-XYINQOWIXKZRXPOX - XX:XX
:KUBE-SVC-XXTWNWKYZXOYZLIG - XX:XX
:KUBE-SVC-XXMTJXRTCKKYXXVU - XX:XX
:KUBE-SVC-XEXXYJTQXQXQNGZV - XX:XX
:KUBE-SVC-XJXIWXXNXHJBRVXX - XX:XX
:KUBE-SVC-XJDXXICUXVGNQXQD - XX:XX
:KUBE-SVC-XSZCKCPQDKNXDFNS - XX:XX
:KUBE-SVC-XYOTXXXMXXXTHYYW - XX:XX
:KUBE-SVC-XXXZXJXKCHUXXMXN - XX:XX
:KUBE-SVC-XOWGKPDATKXHHPYT - XX:XX
:KUBE-SVC-XPAMRSXFYTJJJSYX - XX:XX
:KUBE-SVC-XRJIKSAYLIBAEXTX - XX:XX
:KUBE-SVC-XTTMRMIXNXXOMXZK - XX:XX
:KUBE-SVC-XXOCCNLMHHXQRKSY - XX:XX
:KUBE-SVC-XAKOWCXINCDVIIMX - XX:XX
:KUBE-SVC-XFURLXOTYJGXIXGM - XX:XX
:KUBE-SVC-XHZXWFZUXQFDPKQM - XX:XX
:KUBE-SVC-XOQXXQBKMUVXZZHX - XX:XX
:KUBE-SVC-XQXJSKEMIHWVHPEU - XX:XX
:KUBE-SVC-XRKGQQLXKBIGCWPQ - XX:XX
:KUBE-SVC-XWIAXGLGISJXPSYX - XX:XX
:KUBE-SVC-XXXKLGXMSXTLAAXH - XX:XX
:KUBE-SVC-XXXZXGUBIBKCXMVX - XX:XX
:KUBE-SVC-XXSMIBXFSYXEJIXR - XX:XX
:KUBE-SVC-XANHUZMJHMXAUHCX - XX:XX
:KUBE-SVC-XDBXKXGSUZLPGKXN - XX:XX
:KUBE-SVC-XIIGIURZARRLGVAQ - XX:XX
:KUBE-SVC-XJXGXXXKXVPPNYDX - XX:XX
:KUBE-SVC-XKMSFIPLSOXXYHMF - XX:XX
:KUBE-SVC-XKQXCLOXSVHXESIC - XX:XX
:KUBE-SVC-XKUBSMFPMNCCWXAG - XX:XX
:KUBE-SVC-XMXNYMRVVMJIUWGB - XX:XX
:KUBE-SVC-XNXCDXLNPCFXXGLT - XX:XX
:KUBE-SVC-XOXRXBPGZXTINJQJ - XX:XX
:KUBE-SVC-XPEXXFXXDMVDXEGX - XX:XX
:KUBE-SVC-AXGHWLTXXIMXTKXK - XX:XX
:KUBE-SVC-ACGXJWXAEQISTIOS - XX:XX
:KUBE-SVC-AGQEAXDEKHQGCFOU - XX:XX
:KUBE-SVC-AHXXZIIRSEPXPIKX - XX:XX
:KUBE-SVC-AICNZLUXHMXNXBXV - XX:XX
:KUBE-SVC-ALUGOXTPGIKZDKZU - XX:XX
:KUBE-SVC-ARZRXWKUZUHPZXPH - XX:XX
:KUBE-SVC-ASJYXHXTFCQVNXQF - XX:XX
:KUBE-SVC-AXDKZXSCYUKMDXYX - XX:XX
:KUBE-SVC-BXUDJOXXJHVJOHHE - XX:XX
:KUBE-SVC-BBZSSVXFXYEXQTGN - XX:XX
:KUBE-SVC-BGWYTXCLUEZQFXTP - XX:XX
:KUBE-SVC-BLZBCBNXVRQXMIXZ - XX:XX
:KUBE-SVC-BNQERAVTXGWNPVPB - XX:XX
:KUBE-SVC-BPBBFYXJUBAEXDND - XX:XX
:KUBE-SVC-BUGAXNWNXDNZXNOO - XX:XX
:KUBE-SVC-BXORXHXWNXZBSYXG - XX:XX
:KUBE-SVC-CXNJQJHXAUHNICZL - XX:XX
:KUBE-SVC-CFQYBBCIXXHBFTMC - XX:XX
:KUBE-SVC-CHRTKFCEFVMLLAOQ - XX:XX
:KUBE-SVC-CLDVMJSPGLBDHHNW - XX:XX
:KUBE-SVC-CTTFXKEXXTXKBSOB - XX:XX
:KUBE-SVC-CUCLTQZFSLSSNMAB - XX:XX
:KUBE-SVC-CUHLIXXZGUXKWGUQ - XX:XX
:KUBE-SVC-CXSBXXRXVXFGGXDH - XX:XX
:KUBE-SVC-DXLKXPCZKQXDXHQG - XX:XX
:KUBE-SVC-DBDXFMFHXXLBJXXC - XX:XX
:KUBE-SVC-DECQCLSHXXHXVFUM - XX:XX
:KUBE-SVC-DGQOXKXGNXWXRSGK - XX:XX
:KUBE-SVC-DMAQJBLZXCXGOIXJ - XX:XX
:KUBE-SVC-DPYMXXXMXQBXZQBY - XX:XX
:KUBE-SVC-DRVTNYINXDICIJLV - XX:XX
:KUBE-SVC-EXXFEJXTPSMZKHXS - XX:XX
:KUBE-SVC-EXXAWQCXKMJSLTWG - XX:XX
:KUBE-SVC-EXXXWXUXJXIJYRCX - XX:XX
:KUBE-SVC-EIDXXXJUJXEXDLWF - XX:XX
:KUBE-SVC-ENDXXMLXICOZCDBN - XX:XX
:KUBE-SVC-ETKFQRWWWGDOOZXX - XX:XX
:KUBE-SVC-EUXORBTXHFRIXGXJ - XX:XX
:KUBE-SVC-EUFAGUTGVTRLBVXA - XX:XX
:KUBE-SVC-EVNLDBXXEEALUPVJ - XX:XX
:KUBE-SVC-EXLCBKKQPWVSSXCX - XX:XX
:KUBE-SVC-FXFACXTYJUXXBTXW - XX:XX
:KUBE-SVC-FXPOFXXXTGXCWNMX - XX:XX
:KUBE-SVC-FADXWXPXVRGFGXNK - XX:XX
:KUBE-SVC-FBXIWUMANQGXXQXR - XX:XX
:KUBE-SVC-FCXCZYXPWZFJWQPS - XX:XX
:KUBE-SVC-FCNNXXBSOGILXJZT - XX:XX
:KUBE-SVC-FGSYZTMXXXAPVYYV - XX:XX
:KUBE-SVC-FRPIDXWEEJQFXSRQ - XX:XX
:KUBE-SVC-FSXXDXJTUHHJSOGX - XX:XX
:KUBE-SVC-FSOXNXXGBKQZCBXO - XX:XX
:KUBE-SVC-FTOXJRXWGGBXAXXY - XX:XX
:KUBE-SVC-GXUXWSWNZPMXBWIL - XX:XX
:KUBE-SVC-GLMGVCLJGXAMVQKL - XX:XX
:KUBE-SVC-GMVDXGTFUDJCIXNN - XX:XX
:KUBE-SVC-GSVDZGXEGXJXLCLS - XX:XX
:KUBE-SVC-GZXSVSXXJQUIFLBW - XX:XX
:KUBE-SVC-HXEAXJVCUXLHTQHX - XX:XX
:KUBE-SVC-HXWWXMSYXBPPXBBI - XX:XX
:KUBE-SVC-HXBJECVXSXFAAZUM - XX:XX
:KUBE-SVC-HXOBXZXBOZXXHKXL - XX:XX
:KUBE-SVC-HXVAEEFSXVNKZBGX - XX:XX
:KUBE-SVC-HXXLWLHBNHAAXXAJ - XX:XX
:KUBE-SVC-HCUCOIQKEBEXSKGA - XX:XX
:KUBE-SVC-HFKXCXHXRIVDXPBF - XX:XX
:KUBE-SVC-HFSWQPRTPAYXOXUD - XX:XX
:KUBE-SVC-HIEPXXBXKFWIZAVE - XX:XX
:KUBE-SVC-HNARAAYRXVOXMGQX - XX:XX
:KUBE-SVC-HSHATXXRTQXUIARR - XX:XX
:KUBE-SVC-HTCFDVGOHITFJWIC - XX:XX
:KUBE-SVC-HURETYBXPFFJHQXN - XX:XX
:KUBE-SVC-HWPEQXXGXJXXXXMW - XX:XX
:KUBE-SVC-HWVEXWXITKXWJXJF - XX:XX
:KUBE-SVC-HXEIXSWSWQTIQIGW - XX:XX
:KUBE-SVC-HYSKHXNAXXFLEKEX - XX:XX
:KUBE-SVC-IXOBPOCXDNVHLGXZ - XX:XX
:KUBE-SVC-IXXXCSONBWKXSHJX - XX:XX
:KUBE-SVC-IXXXYOZSRXHJMCIV - XX:XX
:KUBE-SVC-IXOMXMCANYROQEAU - XX:XX
:KUBE-SVC-ICPIXTXGWZNXCXWY - XX:XX
:KUBE-SVC-IKRDPXFVXXNZVXHM - XX:XX
:KUBE-SVC-IMJRWTGXWIXGXIXM - XX:XX
:KUBE-SVC-IRMXFKKXZXXVLXXU - XX:XX
:KUBE-SVC-IWSXXLFASGNXTXXH - XX:XX
:KUBE-SVC-JXAYLUQEXBLBWONN - XX:XX
:KUBE-SVC-JXJOXPJFEXGEDUET - XX:XX
:KUBE-SVC-JXOPJBELBBRTOFNE - XX:XX
:KUBE-SVC-JXZRMVVZTIAJBHNQ - XX:XX
:KUBE-SVC-JXYUGXLXAXXAPFWQ - XX:XX
:KUBE-SVC-JDIJTQXZLHFZKQJB - XX:XX
:KUBE-SVC-JEVUQIBXQIOXIHLG - XX:XX
:KUBE-SVC-JFSRBXUHXPJFKKPC - XX:XX
:KUBE-SVC-JMGWXXPMHVGGOVNX - XX:XX
:KUBE-SVC-JMQXOSEYEXBLMEXD - XX:XX
:KUBE-SVC-JQTIXPXXCHQAQVMX - XX:XX
:KUBE-SVC-JRPXRIORNTHOXXXD - XX:XX
:KUBE-SVC-KDGKRHAXXXGRNXXX - XX:XX
:KUBE-SVC-KIXYXKVWPIWNUXQP - XX:XX
:KUBE-SVC-KLXNVPJORZNUZIXX - XX:XX
:KUBE-SVC-KQUCXEVGZPBXQZXH - XX:XX
:KUBE-SVC-KVQKXXPKTGXFEVSF - XX:XX
:KUBE-SVC-LXPIFDXJEXHYXDEV - XX:XX
:KUBE-SVC-LXDUARETTKXEOQOD - XX:XX
:KUBE-SVC-LXIIXLJXFBXEJXMS - XX:XX
:KUBE-SVC-LCQXWTXWEMZGXZXM - XX:XX
:KUBE-SVC-LCTOAEXJXXAPQHVZ - XX:XX
:KUBE-SVC-LGXXXWRDLXBYZXXX - XX:XX
:KUBE-SVC-LHNNXYBZAAJXXCXN - XX:XX
:KUBE-SVC-LKXQVTQXPMBUEXCU - XX:XX
:KUBE-SVC-LSJKXKPVQBXMANAL - XX:XX
:KUBE-SVC-LUFMPUTDZOXTCBXV - XX:XX
:KUBE-SVC-LVXJYIMGFNPIFZQB - XX:XX
:KUBE-SVC-MXXFXGXXQOJCNQXG - XX:XX
:KUBE-SVC-MBGYWZNVXKQSNXQP - XX:XX
:KUBE-SVC-MDXQDSJFMWZGNAKX - XX:XX
:KUBE-SVC-MEMLUNAXXIXIDXGX - XX:XX
:KUBE-SVC-MEUIGUHXHTOXLTXA - XX:XX
:KUBE-SVC-MGTVLWBXCGLMTOAX - XX:XX
:KUBE-SVC-MYQLZXIXODHNIXXY - XX:XX
:KUBE-SVC-NXGUSAELHXURYEXX - XX:XX
:KUBE-SVC-NJLZPFFFNZXPXGAH - XX:XX
:KUBE-SVC-NORIXMEBTFXKXVVN - XX:XX
:KUBE-SVC-NPXXXMXPTMTKRNXY - XX:XX
:KUBE-SVC-NQZPYGXSXZXJXJAV - XX:XX
:KUBE-SVC-NRAVLGXVZBYXOXDP - XX:XX
:KUBE-SVC-NWUXXNQRGOFTYRVA - XX:XX
:KUBE-SVC-NXBZKXFOOJLYMAKW - XX:XX
:KUBE-SVC-NZPXYXZEJXVXXWTL - XX:XX
:KUBE-SVC-OXBXWXXSCGGKQGMG - XX:XX
:KUBE-SVC-OXRXDXUXAANXTOXX - XX:XX
:KUBE-SVC-OXUEESQXMXZCXTLV - XX:XX
:KUBE-SVC-OBXMXXREGSXPRCVX - XX:XX
:KUBE-SVC-OFHLXJXNWGCJNZEY - XX:XX
:KUBE-SVC-OHKMZXJVXYXGRXGZ - XX:XX
:KUBE-SVC-OIXXXIXGAYANIUNY - XX:XX
:KUBE-SVC-OMKOKTXOFHUGWLJN - XX:XX
:KUBE-SVC-ONMXRXOXHRXCOXGH - XX:XX
:KUBE-SVC-OXGEUBSAOXPWSXTF - XX:XX
:KUBE-SVC-OZGQFFEXIIXHZXBJ - XX:XX
:KUBE-SVC-PXRUSNTZUGGXOFXX - XX:XX
:KUBE-SVC-PXLBBYXEXXXDSXHM - XX:XX
:KUBE-SVC-PBMZKPRXSXPWXNQG - XX:XX
:KUBE-SVC-PDXYXXUOWQMEIXXE - XX:XX
:KUBE-SVC-PEWTBIVHUXXSOFYV - XX:XX
:KUBE-SVC-PNRPNORAEXDKHKTY - XX:XX
:KUBE-SVC-PUXGQXWTKXVEZXWE - XX:XX
:KUBE-SVC-QXLXSXPODCNPORXU - XX:XX
:KUBE-SVC-QXPLXPGLXRCUPXWG - XX:XX
:KUBE-SVC-QBZOPXKIXXBWRUXB - XX:XX
:KUBE-SVC-QFDJBTXXHXVFXQXB - XX:XX
:KUBE-SVC-QGQUXGAQXHKLJXGY - XX:XX
:KUBE-SVC-QISOODXAQKJMISXT - XX:XX
:KUBE-SVC-QKVXBXQYQSSXXOFM - XX:XX
:KUBE-SVC-QRTPUHKXXCZKXDCR - XX:XX
:KUBE-SVC-QWXBEWKEUXPNWITP - XX:XX
:KUBE-SVC-RXXEXKNXHXSXOLXX - XX:XX
:KUBE-SVC-RXXJODPXIRXWGTBE - XX:XX
:KUBE-SVC-RXNHSUXKIXXPGTVU - XX:XX
:KUBE-SVC-RBRSHHXAXXQKWEBR - XX:XX
:KUBE-SVC-RCXXHARMXHQXWYCY - XX:XX
:KUBE-SVC-RCXMYQNMDOPXVQBX - XX:XX
:KUBE-SVC-RGOXTPKGXYZXXXVJ - XX:XX
:KUBE-SVC-RNXERQXBEXZDIJQX - XX:XX
:KUBE-SVC-RRRTXYQBWWKJDBGX - XX:XX
:KUBE-SVC-RVQBCAXUXVCVZORX - XX:XX
:KUBE-SVC-SXGNXXRFSNZJQGBX - XX:XX
:KUBE-SVC-SFGNJEIRRIUXMFRW - XX:XX
:KUBE-SVC-SMXXRRPEUKMAAQXX - XX:XX
:KUBE-SVC-SXCPBCXYGRYYSULL - XX:XX
:KUBE-SVC-TXGRNXIHRFVEYXPT - XX:XX
:KUBE-SVC-TXXGQXCJOMVOZBBX - XX:XX
:KUBE-SVC-TXIJAXWADCNXVDYU - XX:XX
:KUBE-SVC-TXXAGXXTIOKPXUXV - XX:XX
:KUBE-SVC-TBXCYTQSHMMKCABN - XX:XX
:KUBE-SVC-TIGHJYAXIGXLXQUB - XX:XX
:KUBE-SVC-TMSWZXXUSXHGQCEC - XX:XX
:KUBE-SVC-UXWPUXBMDXIXXVXR - XX:XX
:KUBE-SVC-UXLGJYXXAOEXAXCN - XX:XX
:KUBE-SVC-UXVXGQPGYBXCXHBA - XX:XX
:KUBE-SVC-UCKRSXVFXLWRBQXX - XX:XX
:KUBE-SVC-UIPBDKQXOEIOTXXU - XX:XX
:KUBE-SVC-UMDLOONOEFMGBPAY - XX:XX
:KUBE-SVC-URRVHCUSXHVBWZEX - XX:XX
:KUBE-SVC-VXDLMYAXKWVXUTXX - XX:XX
:KUBE-SVC-VFADXEYXZKNUGSEX - XX:XX
:KUBE-SVC-VKLGXKCXGMVXWUEB - XX:XX
:KUBE-SVC-VMMVURAXDBXXXXJN - XX:XX
:KUBE-SVC-VPVJXGTKLPDXGBXM - XX:XX
:KUBE-SVC-VSXXRGEHXXNYHXFJ - XX:XX
:KUBE-SVC-VYXOQXXZWIXUPHVS - XX:XX
:KUBE-SVC-WXXITJBXZTXLRQTL - XX:XX
:KUBE-SVC-WXTSXSXXXJTYQTXC - XX:XX
:KUBE-SVC-WXUIXXNJXDGCJDNE - XX:XX
:KUBE-SVC-WXIKJOVUDXGZSFCY - XX:XX
:KUBE-SVC-WGZMXNSXKCTBXXHX - XX:XX
:KUBE-SVC-WJXRTRAQDWVXWXUW - XX:XX
:KUBE-SVC-WNTRXXZETDXWAXXG - XX:XX
:KUBE-SVC-WQNUPYEZVKLVRPCZ - XX:XX
:KUBE-SVC-WUGXWRBAXOXANDDK - XX:XX
:KUBE-SVC-WXXGPAQXELITGJRJ - XX:XX
:KUBE-SVC-WYXLXLXXSMXXFHNL - XX:XX
:KUBE-SVC-WZXXKXXTYKVIQOXN - XX:XX
:KUBE-SVC-XXNQGLSXEOXQXDXF - XX:XX
:KUBE-SVC-XXHWYXEXXXMGKHYU - XX:XX
:KUBE-SVC-XXPJXGBBHJKAFHFG - XX:XX
:KUBE-SVC-XCQXYGSPTQXPXXXN - XX:XX
:KUBE-SVC-XCXNXMCRNAULYXWN - XX:XX
:KUBE-SVC-XDXVZTEZJCXVWGTQ - XX:XX
:KUBE-SVC-XHXPAAZVXBMVXALJ - XX:XX
:KUBE-SVC-XHTBCBWAAQTDXIXJ - XX:XX
:KUBE-SVC-XLYZVSOKYJUXXOIP - XX:XX
:KUBE-SVC-XOWEHFXSGCMCXBXX - XX:XX
:KUBE-SVC-XQXGSAEXPJFQYRJQ - XX:XX
:KUBE-SVC-XRAYXJPZXMFQXSRX - XX:XX
:KUBE-SVC-XVKLXNZPBXXLXZDU - XX:XX
:KUBE-SVC-XYIXQRQCWSXADXLG - XX:XX
:KUBE-SVC-YXYBMOOOXHKDLTCX - XX:XX
:KUBE-SVC-YDRXHBXMXDXLOXVE - XX:XX
:KUBE-SVC-YDXANXNUOSFKKXKW - XX:XX
:KUBE-SVC-YFXARXCQXCFBDEKB - XX:XX
:KUBE-SVC-YNXRMMOXVSXWAEVD - XX:XX
:KUBE-SVC-YNMLSRXXMXCNMJXT - XX:XX
:KUBE-SVC-YUGSJKVOJTGRXBOL - XX:XX
:KUBE-SVC-YVUMXXXHOGFYBHPU - XX:XX
:KUBE-SVC-YYGHYHVRZXNERXXX - XX:XX
:KUBE-SVC-ZXXXVMHUFXXRRHZA - XX:XX
:KUBE-SVC-ZXCEXHGUXXFBIXVI - XX:XX
:KUBE-SVC-ZBDQRWJXJGOZVZJL - XX:XX
:KUBE-SVC-ZCXXTYSUJJEZXXXF - XX:XX
:KUBE-SVC-ZGZMOXKMXFCGXXKX - XX:XX
:KUBE-SVC-ZIFWATGFWIJZQKXI - XX:XX
:KUBE-SVC-ZLAOWJUEHQXXTZFK - XX:XX
:KUBE-SVC-ZLINXJCXIHPXTXPX - XX:XX
:KUBE-SVC-ZLSRFQYMOZMAZXSP - XX:XX
:KUBE-SVC-ZQJNYVXRXXOWSDOS - XX:XX
:KUBE-SVC-ZSUHDTKEKQDANICV - XX:XX
:KUBE-SVC-ZTAHJCVAMDTXDLDZ - XX:XX
:KUBE-SVC-ZUMKTXFQRYYYIPCX - XX:XX
-A PREROUTING -m comment --comment "comment" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "comment" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "comment" -j KUBE-POSTROUTING
-A POSTROUTING -s XXX.XXX.XXX.X/XX -j FLANNEL
-A POSTROUTING ! -s XXX.XXX.XXX.X/XX -d XXX.XXX.XXX.X/XX -j MASQUERADE
-A FLANNEL -d XXX.XXX.XXX.X/XX -j ACCEPT
-A FLANNEL ! -d XXX.X.X.X/X -j MASQUERADE
-A KUBE-MARK-DROP -j MARK --set-xmark XxXXXX/XxXXXX
-A KUBE-MARK-DROP -j MARK --set-xmark XxXXXX/XxXXXX
-A KUBE-MARK-DROP -j MARK --set-xmark XxXXXX/XxXXXX
-A KUBE-MARK-DROP -j MARK --set-xmark XxXXXX/XxXXXX
-A KUBE-MARK-MASQ -j MARK --set-xmark XxXXXX/XxXXXX
-A KUBE-POSTROUTING -m comment --comment "comment" -m mark --mark XxXXXX/XxXXXX -j MASQUERADE
-A KUBE-SEP-XXDIJNCXZNXRXCDK -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXDIJNCXZNXRXCDK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXNKLXKTWBXXGGRM -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXNKLXKTWBXXGGRM -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXOOEPTHROQWXXZB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXOOEPTHROQWXXZB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XGCOHXVPDXTXIZUS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XGCOHXVPDXTXIZUS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XIIVCPFXKMCXHWBX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XIIVCPFXKMCXHWBX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XMIJSFZXSOXERHCE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XMIJSFZXSOXERHCE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XTJZYLXVVPCOXLWH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XTJZYLXVVPCOXLWH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXJNSHVLAXXBZXYX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXJNSHVLAXXBZXYX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XGWLAZBOXUKXFXKX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XGWLAZBOXUKXFXKX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XIAXOXIVJMCHKMXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XIAXOXIVJMCHKMXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XKNYFWSSIXNVVCPB -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XKNYFWSSIXNVVCPB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XLTTNSWMTXQCZQTJ -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XLTTNSWMTXQCZQTJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XOXOKVYPYXKTUSSB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XOXOKVYPYXKTUSSB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XVREBNCXQXVXUHUW -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XVREBNCXQXVXUHUW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXRXZAXCZYXDZMMA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXRXZAXCZYXDZMMA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XYSFKYVNDXOUESZX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XYSFKYVNDXOUESZX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXCSXBXVBPXHXHVE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXCSXBXVBPXHXHVE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXLXYXSXXXYXQXYX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXLXYXSXXXYXQXYX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXBNXAGOXXXFKJYX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXBNXAGOXXXFKJYX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXDXHIGQIZHPOBEL -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXDXHIGQIZHPOBEL -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXZBGBXOJKUEVCXO -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXZBGBXOJKUEVCXO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXIXWXTXXZWQYCUF -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXIXWXTXXZWQYCUF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XAXMWFMJJXXXGWJU -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XAXMWFMJJXXXGWJU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XASJXLVYYABDOXRR -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XASJXLVYYABDOXRR -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XBFJMUPXHAZLXXXE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XBFJMUPXHAZLXXXE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XBGXPXYXLKZVXXIA -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XBGXPXYXLKZVXXIA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XHRAZUVCRXGRYEXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XHRAZUVCRXGRYEXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XIAZXNFOJXDGWPXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XIAZXNFOJXDGWPXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XINDJEXBGAFNKMDK -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XINDJEXBGAFNKMDK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XIXCIFAXDJVZTSRL -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XIXCIFAXDJVZTSRL -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XRCMSDHCNWFSGCZG -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XRCMSDHCNWFSGCZG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XRHHCDXTPWEGIXLX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XRHHCDXTPWEGIXLX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XRZGCKXKVSUWTXVX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XRZGCKXKVSUWTXVX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XVRXIRHFHURMXKXB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XVRXIRHFHURMXKXB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXODXIDXOHNUJZWZ -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXODXIDXOHNUJZWZ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXPJZHXOJGZJFDYA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXPJZHXOJGZJFDYA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XAHRYQUUXYXIDXSI -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XAHRYQUUXYXIDXSI -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XGXYXOQECSCHYQSM -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XGXYXOQECSCHYQSM -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XIQWDXXBCXVOXZXB -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XIQWDXXBCXVOXZXB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XKNFNNDXSFQKMGDR -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XKNFNNDXSFQKMGDR -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XMOMWXXUCGJWUDWX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XMOMWXXUCGJWUDWX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XNAAXXOPTCMXKXYM -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XNAAXXOPTCMXKXYM -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XNFXZXYXWDJZKDLD -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XNFXZXYXWDJZKDLD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XRPQZGHSFAJEXXIP -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XRPQZGHSFAJEXXIP -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XWPBVGXXTDFXFYDG -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XWPBVGXXTDFXFYDG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXLXWSLUXFXTXSXT -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXLXWSLUXFXTXSXT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXKCVBUXXXLZXVQG -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXKCVBUXXXLZXVQG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXXXILOCXGNBWYKX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXXXILOCXGNBWYKX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XBGYCPBSKXXKXOOJ -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XBGYCPBSKXXKXOOJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XBIXEZSSCVBXOTPG -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XBIXEZSSCVBXOTPG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XCFAFXXPXUXXNXXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XCFAFXXPXUXXNXXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XGTIANXWGTJQCXNE -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XGTIANXWGTJQCXNE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XHFUTFXXXOSXVVHH -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XHFUTFXXXOSXVVHH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XIXRRGOIWSTKXYSX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XIXRRGOIWSTKXYSX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XSJLKSXTXXCQXCXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XSJLKSXTXXCQXCXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XZXXWIXGPGHXFHSX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XZXXWIXGPGHXFHSX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXIONLXMLGFZOZFB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXIONLXMLGFZOZFB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XXPIAXSAIFUHQGND -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXPIAXSAIFUHQGND -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XAXPXGVMAZEXPTBX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XAXPXGVMAZEXPTBX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XBKXXGQQCWXLAXYD -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XBKXXGQQCWXLAXYD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XCJATFMBWNQXFXNS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XCJATFMBWNQXFXNS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XGQXNUXUVVREFJGE -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XGQXNUXUVVREFJGE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XHXSXRXVRXXXFOUP -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XHXSXRXVRXXXFOUP -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XHJPKSWXXNYXZXXC -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XHJPKSWXXNYXZXXC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XPNHQHILSTXODFPD -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XPNHQHILSTXODFPD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XPUCLUJXGNJMQURT -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XPUCLUJXGNJMQURT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XSXIXXXPHIHIPXLE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XSXIXXXPHIHIPXLE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XUCFXAHUXAXXXALA -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XUCFXAHUXAXXXALA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XUJHIJKZBXHGAKXS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XUJHIJKZBXHGAKXS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XZXRBDLLPAKXIPZD -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XZXRBDLLPAKXIPZD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-AEXDRVALKCXYXGMW -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AEXDRVALKCXYXGMW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-AEVCABFEIEGRXTEK -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AEVCABFEIEGRXTEK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-AHAFXLXAGUXXQUXP -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AHAFXLXAGUXXQUXP -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-AJXXRTXONUXGZJXT -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AJXXRTXONUXGZJXT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-AJFMIXBSJXVEXXXT -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AJFMIXBSJXVEXXXT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-AKXTJHXIXXXYKOXE -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AKXTJHXIXXXYKOXE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-AKQXXGVYESGFXLFO -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AKQXXGVYESGFXLFO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ALFFLJPXKDVXNTNX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ALFFLJPXKDVXNTNX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-AMGCZYBIAHHUXXXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AMGCZYBIAHHUXXXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-AMJEXFSWNSJLDDAC -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AMJEXFSWNSJLDDAC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-APXHMALBXDXHLZCX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-APXHMALBXDXHLZCX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-AROFZVLFCAODIYUK -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-AROFZVLFCAODIYUK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-ATXKFPRQXCTRNXSE -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ATXKFPRQXCTRNXSE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-BXXBQRXTLFXNXDJX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-BXXBQRXTLFXNXDJX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-BHRJDNOQXXZKRTUC -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-BHRJDNOQXXZKRTUC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-BLMPNFXAIDYJSKXO -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-BLMPNFXAIDYJSKXO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-BRDXGGFXXYBFPSJT -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-BRDXGGFXXYBFPSJT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-BYGNXIVTCZNXXXNV -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-BYGNXIVTCZNXXXNV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-CXXZIBLLCLRNKDXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CXXZIBLLCLRNKDXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-CBXTTPIDEXKXXNEV -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CBXTTPIDEXKXXNEV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-CDBMXXGNMILHQLXV -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CDBMXXGNMILHQLXV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-CEOAQHQEICXMXXHD -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CEOAQHQEICXMXXHD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-CIQZXXPZXWDZXOAK -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CIQZXXPZXWDZXOAK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-CPCDSSDEDRQBVJCV -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CPCDSSDEDRQBVJCV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-CXXLQHOXTXTRPOVG -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CXXLQHOXTXTRPOVG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-CXXXSQXWXXXJBIYX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CXXXSQXWXXXJBIYX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-CZVAWNRXDWMSIVNA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-CZVAWNRXDWMSIVNA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-DXJVGTXXONKLONXZ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DXJVGTXXONKLONXZ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-DXIZHYZXQEXBXXYQ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DXIZHYZXQEXBXXYQ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-DBMRXUMTBXVRSJXD -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DBMRXUMTBXVRSJXD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-DKXMXFNXTBESXDLE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DKXMXFNXTBESXDLE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-DMNYWOPVWRDKXOTX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DMNYWOPVWRDKXOTX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-DNMECZZXLFQXXIMD -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DNMECZZXLFQXXIMD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-DUUXNYOZROGKKSIM -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DUUXNYOZROGKKSIM -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-DXXFXHFXXLXZOBGD -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-DXXFXHFXXLXZOBGD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-EXFOQXDFXSZXZIDW -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EXFOQXDFXSZXZIDW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-EXXNBXDNWCWGTXQT -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EXXNBXDNWCWGTXQT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-EDASXVXXMMUKUEXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EDASXVXXMMUKUEXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-EMSIJAKTGRDEXXGJ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EMSIJAKTGRDEXXGJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-EQOLXXXHXOXKXSHX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EQOLXXXHXOXKXSHX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ERXMCXSJXRQRXOXE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ERXMCXSJXRQRXOXE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ESSREXJUXFEYEUTH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ESSREXJUXFEYEUTH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ESSSFRXQUDXIXXQC -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ESSSFRXQUDXIXXQC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ESYUEXZIXMXXOZUX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ESYUEXZIXMXXOZUX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-EVXXCFGMAOUPYJRU -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EVXXCFGMAOUPYJRU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-EWKXSTBCSZXXHBYH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKXSTBCSZXXHBYH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-EXDXURVOXUWSRDXD -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-EXDXURVOXUWSRDXD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-FXXVXQJVXXXZMMEX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FXXVXQJVXXXZMMEX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-FCNMQLICDXZXHGPS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FCNMQLICDXZXHGPS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-FDFXZYXLXZOXUNRB -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FDFXZYXLXZOXUNRB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-FEDZSXKERFEDKMDY -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FEDZSXKERFEDKMDY -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-FFLSSXXXCURPOXJJ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FFLSSXXXCURPOXJJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-FMBHPSBNXPJHFQMN -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FMBHPSBNXPJHFQMN -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-FNQXHMYFXFGLPUXC -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FNQXHMYFXFGLPUXC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-FOCKHSOXXYZXGXMQ -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FOCKHSOXXYZXGXMQ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-FQXVOLQGTXXBXUXN -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FQXVOLQGTXXBXUXN -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-FTDXXXXXGPTBZRVX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FTDXXXXXGPTBZRVX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-FTSYCICMKRQYJBZH -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FTSYCICMKRQYJBZH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-FWTRXOFKSAINOBXB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-FWTRXOFKSAINOBXB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-GXXRIGDZOHBSSXPX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GXXRIGDZOHBSSXPX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-GXXZXMQXOXAOJXBS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GXXZXMQXOXAOJXBS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-GACXVHRXIXXNCYSG -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GACXVHRXIXXNCYSG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-GAOPRXWQBCCEKBON -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GAOPRXWQBCCEKBON -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-GCUUDEUAQBSJXNNX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GCUUDEUAQBSJXNNX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-GHGUXUSZFMXIMXIB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GHGUXUSZFMXIMXIB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-GIDXXMFAPLTJTTVV -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GIDXXMFAPLTJTTVV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-GKAYASHWKXGHUSFX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GKAYASHWKXGHUSFX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-GPUXOXSQSFYNYWXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GPUXOXSQSFYNYWXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-GQXKJUBTXXZXSPQV -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GQXKJUBTXXZXSPQV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-GXXGWXLLFCXLDXDF -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-GXXGWXLLFCXLDXDF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-HXJCXHXXOTYFZRXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-HXJCXHXXOTYFZRXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-HXXRYXGSAACJKMQD -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-HXXRYXGSAACJKMQD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-HATXEDGCLVXINEXH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-HATXEDGCLVXINEXH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-HGIVSFRTXXAXNHXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-HGIVSFRTXXAXNHXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-HSFDSOYCXLBMRHXT -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-HSFDSOYCXLBMRHXT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-HYEKGUMHLPORRXFX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-HYEKGUMHLPORRXFX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-HYQKNCVLILQXQYQY -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-HYQKNCVLILQXQYQY -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-IXVXPLSXZBBVDACE -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IXVXPLSXZBBVDACE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-IXGKMHTPNPCKXDQF -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IXGKMHTPNPCKXDQF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-IDXXIXVVUPCWOZBX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IDXXIXVVUPCWOZBX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-IFXCFXFXYZSDGXDC -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IFXCFXFXYZSDGXDC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-IMXOZNUHPVKNJOCG -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IMXOZNUHPVKNJOCG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-IOXBCMXXSLXSAJXO -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IOXBCMXXSLXSAJXO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-IUXCMFIVXXTXAINK -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IUXCMFIVXXTXAINK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-IXXLVXULXQXXYCKG -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IXXLVXULXQXXYCKG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-IZPCXCXQXRXMHVLX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-IZPCXCXQXRXMHVLX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-JXSXTLBYGWDHBQXS -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-JXSXTLBYGWDHBQXS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-JXZYVTIQGTWNQGWA -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-JXZYVTIQGTWNQGWA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-JFEXFKXXXMXSDMVI -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-JFEXFKXXXMXSDMVI -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-JOBNTVXNLXDAWAXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-JOBNTVXNLXDAWAXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-JPXKXXNXYPXQTZXE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-JPXKXXNXYPXQTZXE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-JRZSWCXXXXDAXQCH -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-JRZSWCXXXXDAXQCH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-JUJXSSMXDGDXUKCX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-JUJXSSMXDGDXUKCX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-KXCBJXXTNTJXTEIX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-KXCBJXXTNTJXTEIX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-KEXDLPXIMJFBYEQX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-KEXDLPXIMJFBYEQX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-KKXXOIXXCXFYGMBH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-KKXXOIXXCXFYGMBH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-KKJKRURKHXHTIRRW -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-KKJKRURKHXHTIRRW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-KLXTPYGHNGBNNDXC -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-KLXTPYGHNGBNNDXC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-KRXXMFEWCAXXLXXF -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-KRXXMFEWCAXXLXXF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-KVZCXXXXQNMSXZZF -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-KVZCXXXXQNMSXZZF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-LXYLUAILFMGAZHGA -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LXYLUAILFMGAZHGA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-LACEFMXTWPWGIOTJ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LACEFMXTWPWGIOTJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-LBOGUSDMBAJJDOXK -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LBOGUSDMBAJJDOXK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-LHBORJVHQFXXDEHT -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LHBORJVHQFXXDEHT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-LHHXGNGPUKWXONLW -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LHHXGNGPUKWXONLW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-LKAXZUEIEXHGXEXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LKAXZUEIEXHGXEXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-LQAALXZAKRVXBRXA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQAALXZAKRVXBRXA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-LYZJOBIRJOXAWBOO -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LYZJOBIRJOXAWBOO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-LZXJDAONXXXBPHXB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-LZXJDAONXXXBPHXB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MXMIWXXDLEUVOERO -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MXMIWXXDLEUVOERO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MXBXXCCFXMSUDXOB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MXBXXCCFXMSUDXOB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MAXXHNMXYXTIXXXH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MAXXHNMXYXTIXXXH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MMFRTYIXXTIXBJQU -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MMFRTYIXXTIXBJQU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MNCXKIDAGVJPJJKO -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MNCXKIDAGVJPJJKO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MOXDXQXIOXBELXWX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MOXDXQXIOXBELXWX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MQEWOJWPHPVOHDNX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MQEWOJWPHPVOHDNX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MRXXFQXXXFCPRDEF -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MRXXFQXXXFCPRDEF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-MXDEFXGZSWCWDDXM -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-MXDEFXGZSWCWDDXM -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-NXXHEXHAJBWZANYE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-NXXHEXHAJBWZANYE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-NIXDSDLDDNYDRCQN -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-NIXDSDLDDNYDRCQN -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-NLSXKXNQEAZULNJX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-NLSXKXNQEAZULNJX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-NMYESYJXHAMBXEVX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-NMYESYJXHAMBXEVX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-NUJRJIJXJSISCXUL -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-NUJRJIJXJSISCXUL -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-NXGXLAEZDXDQCTXJ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-NXGXLAEZDXDQCTXJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-NZIXOBCPMZOIAZFA -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-NZIXOBCPMZOIAZFA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-OXRXPXXDOHCVUAEM -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-OXRXPXXDOHCVUAEM -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-OXNXGYBXXNHVTWXG -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-OXNXGYBXXNHVTWXG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-OIYKXYEXNOXFYXDX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-OIYKXYEXNOXFYXDX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-OLXUXSFXWZAQXQXJ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-OLXUXSFXWZAQXQXJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-OPAVXQXNDINUXESF -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-OPAVXQXNDINUXESF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-OPJXLKWNWPQXVXXH -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-OPJXLKWNWPQXVXXH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-ORLFLCRNYFTCFDXT -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ORLFLCRNYFTCFDXT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-OSKWUDBAZCYEXMXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-OSKWUDBAZCYEXMXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-PXWUXNTCIGZXONUA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PXWUXNTCIGZXONUA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-PXEIHHMPWXRIOTXZ -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PXEIHHMPWXRIOTXZ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-PBKBKAYRMXOXHFLQ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PBKBKAYRMXOXHFLQ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-PDOBQZHDNANTXLZX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PDOBQZHDNANTXLZX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-PFXQQXCBPXFCJOXS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PFXQQXCBPXFCJOXS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-PIEADXBQPJGHEXIW -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PIEADXBQPJGHEXIW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-PPZLJYXXUZXKKXSS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PPZLJYXXUZXKKXSS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-PQFZVDXLXXXCDDHZ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-PQFZVDXLXXXCDDHZ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-QXMKRQTRXXYGNDSU -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-QXMKRQTRXXYGNDSU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-QBJAQUBXHUBBUBXT -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-QBJAQUBXHUBBUBXT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-QECIAEFZDXHXIRNQ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-QECIAEFZDXHXIRNQ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-QJZXXNXDTXZXGYWC -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-QJZXXNXDTXZXGYWC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-QOFMVTGKEPYMXNJX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-QOFMVTGKEPYMXNJX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-QRXUQSXMSFDVZMYZ -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-QRXUQSXMSFDVZMYZ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-RXZZWVXAXAIFMFXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-RXZZWVXAXAIFMFXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-RXVOXXZRPXWXGXGL -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-RXVOXXZRPXWXGXGL -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-RBXHXYAUHDGHENDL -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-RBXHXYAUHDGHENDL -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-REXPQNMGXXAASXPS -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-REXPQNMGXXAASXPS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-RRJHMMXBCKNFAXGU -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-RRJHMMXBCKNFAXGU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-RSXTXXXEJMXPXOXE -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-RSXTXXXEJMXPXOXE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-RTVFTXXXSJDZUTXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-RTVFTXXXSJDZUTXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-SXQXAJMXYMXXXTEX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-SXQXAJMXYMXXXTEX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-SLXJBTMZUNNFRHKS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-SLXJBTMZUNNFRHKS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-STZMJAGTXUPYRKXT -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-STZMJAGTXUPYRKXT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-SUADNDUYKXMQLALM -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-SUADNDUYKXMQLALM -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-SVAQIAHXEXXVXNXP -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-SVAQIAHXEXXVXNXP -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-SZXOYSXJXXQFXXIT -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-SZXOYSXJXXQFXXIT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-TXMKXKFHXZXXWIXK -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TXMKXKFHXZXXWIXK -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-TXHXCMUJUGEXIIVX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TXHXCMUJUGEXIIVX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-TXWRXXLDXUJMPSSU -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TXWRXXLDXUJMPSSU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-TXXYJFUREXXDLLTG -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TXXYJFUREXXDLLTG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-TMXAZNXUSICXXWMB -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TMXAZNXUSICXXWMB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-TOQEXXXPOKTVCDEA -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TOQEXXXPOKTVCDEA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-TPXFEZTUIKWENXLX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TPXFEZTUIKWENXLX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-TVLLXEGQZHXXJBKW -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-TVLLXEGQZHXXJBKW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-UXXPEXMZJDJFNHXX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UXXPEXMZJDJFNHXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-UBYTXZMXSEHXXWXC -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UBYTXZMXSEHXXWXC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-UCXJXFLUUUMRJJYA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UCXJXFLUUUMRJJYA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-UCETASOSIXJDXZXH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UCETASOSIXJDXZXH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-UDXXXXXXTOXMPXZC -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UDXXXXXXTOXMPXZC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-UMGLSKFVXXUSOMTL -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UMGLSKFVXXUSOMTL -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-UONYXWXXRHJALCSS -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UONYXWXXRHJALCSS -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-URUQSXXXPEXDMXTF -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-URUQSXXXPEXDMXTF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-USVXTXXIKXLBNCPB -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-USVXTXXIKXLBNCPB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-UTXOXFNFABWHVGCX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-UTXOXFNFABWHVGCX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-VIAHEXHEXXDPVFUX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VIAHEXHEXXDPVFUX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-VJXNTJXTXJRNWPUD -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VJXNTJXTXJRNWPUD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-VKDULWHHBYJKYKZX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VKDULWHHBYJKYKZX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-VKLMPMSXDXBEISCW -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VKLMPMSXDXBEISCW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-VKXPCXTMFURRPXXX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VKXPCXTMFURRPXXX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-VNSXSXXRGGYEHXXY -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VNSXSXXRGGYEHXXY -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-VSPXRXBFDYRLZIHC -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VSPXRXBFDYRLZIHC -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-VTFOGXMYBZLEXXUT -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VTFOGXMYBZLEXXUT -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-VVNXTXOFJOXIVXXY -s XX.XX.XX.XXX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VVNXTXOFJOXIVXXY -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XX.XX.XX.XXX:XXXX
-A KUBE-SEP-VXXXDZZJXXYXCAXV -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VXXXDZZJXXYXCAXV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-VZSUSPSQYXXCXCXP -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-VZSUSPSQYXXCXCXP -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-WNNUSXXCXBXESMXA -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-WNNUSXXCXBXESMXA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-WWXKKXPXZISFXWYY -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-WWXKKXPXZISFXWYY -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XXNGAFXPKVDCBEOE -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XXNGAFXPKVDCBEOE -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XBIBKZIWRXQFUXIU -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XBIBKZIWRXQFUXIU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XCFYDDJVFZXSQJXU -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XCFYDDJVFZXSQJXU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XGHZLNYDTXGPXVXZ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XGHZLNYDTXGPXVXZ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XHRCRWHRSCXVLUUG -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XHRCRWHRSCXVLUUG -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XJBHGVMWXXSXXJIA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XJBHGVMWXXSXXJIA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XPXNRZRYBXXBWDMI -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XPXNRZRYBXXBWDMI -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-XPYSZLKAPUWWUNKD -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XPYSZLKAPUWWUNKD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XUNLVRURGXZOTLXJ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XUNLVRURGXZOTLXJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-XWDLVELPFVUPQTXV -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-XWDLVELPFVUPQTXV -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-YXXSVELPXQSJXFEB -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-YXXSVELPXQSJXFEB -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXXX
-A KUBE-SEP-YXXUHBBVXRPKUPPY -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-YXXUHBBVXRPKUPPY -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-YXEQXVIXWLOAYPXY -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-YXEQXVIXWLOAYPXY -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-YBIJXQXYCNAAFMSX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-YBIJXQXYCNAAFMSX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-YMVXFQWJTNYIXUQX -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-YMVXFQWJTNYIXUQX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-YPSRQIRBZPXSMDAW -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-YPSRQIRBZPXSMDAW -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-YUXKYZOAIIOLXPGJ -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-YUXKYZOAIIOLXPGJ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-ZXJGHRWWGEGRZTKX -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZXJGHRWWGEGRZTKX -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-ZAMXLSXVYXXBRUKU -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZAMXLSXVYXXBRUKU -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-ZBQPUGNYSXXHSKOO -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZBQPUGNYSXXHSKOO -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ZEXMXXGXLXXAXQJF -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZEXMXXGXLXXAXQJF -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ZFJYNOXNYDZYNUXH -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZFJYNOXNYDZYNUXH -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ZHXAUOTEHBQMELPQ -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZHXAUOTEHBQMELPQ -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ZHDHXBXJLGXDXRXD -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZHDHXBXJLGXDXRXD -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ZHWIXXKLTIJEXKET -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZHWIXXKLTIJEXKET -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SEP-ZPXIDVIKXOVXXCIA -s XXX.XXX.XXX.XX/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZPXIDVIKXOVXXCIA -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.XX:XXXX
-A KUBE-SEP-ZZXRXUIHXXXYXIRL -s XXX.XXX.XXX.X/XX -m comment --comment "comment" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZZXRXUIHXXXYXIRL -p tcp -m comment --comment "comment" -m tcp -j DNAT --to-destination XXX.XXX.XXX.X:XXXX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QBZOPXKIXXBWRUXB
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LVXJYIMGFNPIFZQB
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IXOMXMCANYROQEAU
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HIEPXXBXKFWIZAVE
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XIXXHOZKPQXBJDXF
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XDBXKXGSUZLPGKXN
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-PEWTBIVHUXXSOFYV
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HFKXCXHXRIVDXPBF
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HXVAEEFSXVNKZBGX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-GXUXWSWNZPMXBWIL
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YDXANXNUOSFKKXKW
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EXXFEJXTPSMZKHXS
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-DECQCLSHXXHXVFUM
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FCXCZYXPWZFJWQPS
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZSUHDTKEKQDANICV
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-DBDXFMFHXXLBJXXC
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CXSBXXRXVXFGGXDH
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BUGAXNWNXDNZXNOO
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XOWEHFXSGCMCXBXX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XX -j KUBE-SVC-XKMSFIPLSOXXYHMF
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZQJNYVXRXXOWSDOS
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QXPLXPGLXRCUPXWG
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXSMIBXFSYXEJIXR
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XALVSUXXVMSRVJDD
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HXWWXMSYXBPPXBBI
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LXDUARETTKXEOQOD
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NXGUSAELHXURYEXX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XMXNYMRVVMJIUWGB
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXOCCNLMHHXQRKSY
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-DXLKXPCZKQXDXHQG
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-MYQLZXIXODHNIXXY
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XQXJSKEMIHWVHPEU
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XTTMRMIXNXXOMXZK
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IKRDPXFVXXNZVXHM
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ALUGOXTPGIKZDKZU
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-TXXAGXXTIOKPXUXV
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HCUCOIQKEBEXSKGA
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XQSGXKGXCPLFWVQC
-A KUBE-SERVICES -d XXX.XXX.XXX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-DPYMXXXMXQBXZQBY
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-PUXGQXWTKXVEZXWE
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LKXQVTQXPMBUEXCU
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BXORXHXWNXZBSYXG
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HWVEXWXITKXWJXJF
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HURETYBXPFFJHQXN
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-AXGHWLTXXIMXTKXK
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXWITZOXXXKQZSJN
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CUCLTQZFSLSSNMAB
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XOXRXBPGZXTINJQJ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JFSRBXUHXPJFKKPC
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-VKLGXKCXGMVXWUEB
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XHXPAAZVXBMVXALJ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XANHUZMJHMXAUHCX
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-MEMLUNAXXIXIDXGX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XCQXYGSPTQXPXXXN
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XUNDXXKWWLCXFMXB
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-VMMVURAXDBXXXXJN
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RCXXHARMXHQXWYCY
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CLDVMJSPGLBDHHNW
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NQZPYGXSXZXJXJAV
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JMGWXXPMHVGGOVNX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXTWNWKYZXOYZLIG
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CTTFXKEXXTXKBSOB
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HXOBXZXBOZXXHKXL
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JRPXRIORNTHOXXXD
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-TXXGQXCJOMVOZBBX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JQTIXPXXCHQAQVMX
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XJXGXXXKXVPPNYDX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JXZRMVVZTIAJBHNQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HTCFDVGOHITFJWIC
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CXNJQJHXAUHNICZL
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XOQXXQBKMUVXZZHX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-UMDLOONOEFMGBPAY
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXX -j KUBE-SVC-NPXXXMXPTMTKRNXY
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JEVUQIBXQIOXIHLG
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-AGQEAXDEKHQGCFOU
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QWXBEWKEUXPNWITP
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XNXCDXLNPCFXXGLT
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XDXVZTEZJCXVWGTQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FSXXDXJTUHHJSOGX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ICPIXTXGWZNXCXWY
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-MXXFXGXXQOJCNQXG
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XYIXQRQCWSXADXLG
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-PXLBBYXEXXXDSXHM
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IXXXCSONBWKXSHJX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LSJKXKPVQBXMANAL
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XAXFXWWXXMYXFKAX
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EVNLDBXXEEALUPVJ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-TIGHJYAXIGXLXQUB
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NZPXYXZEJXVXXWTL
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QFDJBTXXHXVFXQXB
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QRTPUHKXXCZKXDCR
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XNWHOQJXZXYDXQAX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XPCRIRAXZQCLXXQO
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XAKOWCXINCDVIIMX
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CUHLIXXZGUXKWGUQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IRMXFKKXZXXVLXXU
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WJXRTRAQDWVXWXUW
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XFURLXOTYJGXIXGM
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXNQGLSXEOXQXDXF
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-AICNZLUXHMXNXBXV
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ARZRXWKUZUHPZXPH
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-KVQKXXPKTGXFEVSF
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXMTJXRTCKKYXXVU
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YNMLSRXXMXCNMJXT
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXXUVXNQMWXYSWTK
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LXIIXLJXFBXEJXMS
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WUGXWRBAXOXANDDK
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-AXDKZXSCYUKMDXYX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZIFWATGFWIJZQKXI
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-KIXYXKVWPIWNUXQP
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HXBJECVXSXFAAZUM
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WXXITJBXZTXLRQTL
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RBRSHHXAXXQKWEBR
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QISOODXAQKJMISXT
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FCNNXXBSOGILXJZT
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FRPIDXWEEJQFXSRQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-TBXCYTQSHMMKCABN
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZTAHJCVAMDTXDLDZ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EXXXWXUXJXIJYRCX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JXOPJBELBBRTOFNE
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZLINXJCXIHPXTXPX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XDDEXMFNHNSQHKLA
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XVLUXXXUGKEGEXJC
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XPAMRSXFYTJJJSYX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XYOTXXXMXXXTHYYW
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OBXMXXREGSXPRCVX
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-GLMGVCLJGXAMVQKL
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IXXXYOZSRXHJMCIV
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WGZMXNSXKCTBXXHX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WXTSXSXXXJTYQTXC
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XHTBCBWAAQTDXIXJ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HWPEQXXGXJXXXXMW
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-SMXXRRPEUKMAAQXX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZGZMOXKMXFCGXXKX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NJLZPFFFNZXPXGAH
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ONMXRXOXHRXCOXGH
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-TXIJAXWADCNXVDYU
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FBXIWUMANQGXXQXR
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HNARAAYRXVOXMGQX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZXXXVMHUFXXRRHZA
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-VSXXRGEHXXNYHXFJ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WXUIXXNJXDGCJDNE
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OXBXWXXSCGGKQGMG
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XAMYINUXQIQJNTKI
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ASJYXHXTFCQVNXQF
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LHNNXYBZAAJXXCXN
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-MEUIGUHXHTOXLTXA
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XRKGQQLXKBIGCWPQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WQNUPYEZVKLVRPCZ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IWSXXLFASGNXTXXH
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-DRVTNYINXDICIJLV
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QKVXBXQYQSSXXOFM
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-UXWPUXBMDXIXXVXR
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BNQERAVTXGWNPVPB
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EXLCBKKQPWVSSXCX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZLSRFQYMOZMAZXSP
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HFSWQPRTPAYXOXUD
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FADXWXPXVRGFGXNK
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OZGQFFEXIIXHZXBJ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-TXGRNXIHRFVEYXPT
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ACGXJWXAEQISTIOS
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BBZSSVXFXYEXQTGN
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FGSYZTMXXXAPVYYV
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-VXDLMYAXKWVXUTXX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ETKFQRWWWGDOOZXX
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XQXGSAEXPJFQYRJQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XJXIWXXNXHJBRVXX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XCMEVYXYGATYYYXX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-MGTVLWBXCGLMTOAX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RNXERQXBEXZDIJQX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-UXVXGQPGYBXCXHBA
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-SXCPBCXYGRYYSULL
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-KQUCXEVGZPBXQZXH
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXXZXJXKCHUXXMXN
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OXUEESQXMXZCXTLV
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WXIKJOVUDXGZSFCY
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XEXXYJTQXQXQNGZV
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OIXXXIXGAYANIUNY
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-MBGYWZNVXKQSNXQP
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RGOXTPKGXYZXXXVJ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WNTRXXZETDXWAXXG
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XKQXCLOXSVHXESIC
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XOWGKPDATKXHHPYT
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IXOBPOCXDNVHLGXZ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-URRVHCUSXHVBWZEX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HXEAXJVCUXLHTQHX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LUFMPUTDZOXTCBXV
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZUMKTXFQRYYYIPCX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXXKLGXMSXTLAAXH
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XSZCKCPQDKNXDFNS
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RVQBCAXUXVCVZORX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YDRXHBXMXDXLOXVE
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XKUBSMFPMNCCWXAG
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BPBBFYXJUBAEXDND
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-PNRPNORAEXDKHKTY
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OMKOKTXOFHUGWLJN
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-AHXXZIIRSEPXPIKX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JMQXOSEYEXBLMEXD
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-SFGNJEIRRIUXMFRW
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WZXXKXXTYKVIQOXN
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXPJXGBBHJKAFHFG
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-UIPBDKQXOEIOTXXU
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RXXJODPXIRXWGTBE
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XSPOKYTTXEMSQXOH
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NRAVLGXVZBYXOXDP
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WYXLXLXXSMXXFHNL
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-GMVDXGTFUDJCIXNN
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LCTOAEXJXXAPQHVZ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RRRTXYQBWWKJDBGX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FTOXJRXWGGBXAXXY
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OXGEUBSAOXPWSXTF
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NXBZKXFOOJLYMAKW
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZBDQRWJXJGOZVZJL
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XYINQOWIXKZRXPOX
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RCXMYQNMDOPXVQBX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BLZBCBNXVRQXMIXZ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LXPIFDXJEXHYXDEV
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XIIGIURZARRLGVAQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZXCEXHGUXXFBIXVI
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-VFADXEYXZKNUGSEX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YUGSJKVOJTGRXBOL
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RXXEXKNXHXSXOLXX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BXUDJOXXJHVJOHHE
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LGXXXWRDLXBYZXXX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-BGWYTXCLUEZQFXTP
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXXX -j KUBE-SVC-XWIAXGLGISJXPSYX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XPEXXFXXDMVDXEGX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZLAOWJUEHQXXTZFK
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XBJXXGHYXBSXSPHX
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ENDXXMLXICOZCDBN
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XAXGXYZSMCBVNXJU
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YFXARXCQXCFBDEKB
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-WXXGPAQXELITGJRJ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-ZCXXTYSUJJEZXXXF
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-VYXOQXXZWIXUPHVS
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-TMSWZXXUSXHGQCEC
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-UXLGJYXXAOEXAXCN
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HSHATXXRTQXUIARR
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JXJOXPJFEXGEDUET
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FXPOFXXXTGXCWNMX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JXYUGXLXAXXAPFWQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XCJBOXYMSYXMXDAT
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EXXAWQCXKMJSLTWG
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-VPVJXGTKLPDXGBXM
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XKAXXYRMXXPFXBND
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-IMJRWTGXWIXGXIXM
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XVKLXNZPBXXLXZDU
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-UCKRSXVFXLWRBQXX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-SXGNXXRFSNZJQGBX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-MDXQDSJFMWZGNAKX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FSOXNXXGBKQZCBXO
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EIDXXXJUJXEXDLWF
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XJDXXICUXVGNQXQD
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-LCQXWTXWEMZGXZXM
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NWUXXNQRGOFTYRVA
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XHZXWFZUXQFDPKQM
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XLYZVSOKYJUXXOIP
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OHKMZXJVXYXGRXGZ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXHWYXEXXXMGKHYU
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XFEADOKDMXXTPSOI
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XCXNXMCRNAULYXWN
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YVUMXXXHOGFYBHPU
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CFQYBBCIXXHBFTMC
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OFHLXJXNWGCJNZEY
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXXZXGUBIBKCXMVX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-CHRTKFCEFVMLLAOQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JDIJTQXZLHFZKQJB
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YXYBMOOOXHKDLTCX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HXXLWLHBNHAAXXAJ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXDXXMLSXXZZXWBX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QGQUXGAQXHKLJXGY
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HYSKHXNAXXFLEKEX
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XRAYXJPZXMFQXSRX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XWJXIXXNJYJDGIXB
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-HXEIXSWSWQTIQIGW
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-PXRUSNTZUGGXOFXX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-PDXYXXUOWQMEIXXE
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YNXRMMOXVSXWAEVD
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-YYGHYHVRZXNERXXX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-KLXNVPJORZNUZIXX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-JXAYLUQEXBLBWONN
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-PBMZKPRXSXPWXNQG
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-DMAQJBLZXCXGOIXJ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EUXORBTXHFRIXGXJ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXNXPPPXLHXVSKXT
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-GSVDZGXEGXJXLCLS
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-GZXSVSXXJQUIFLBW
-A KUBE-SERVICES -d XXX.XXX.XXX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XXX.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XDALDXMRMYUXXXXW
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XRJIKSAYLIBAEXTX
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-OXRXDXUXAANXTOXX
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-XXCEIMYAEEXBXXTZ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-FXFACXTYJUXXBTXW
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-NORIXMEBTFXKXVVN
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-EUFAGUTGVTRLBVXA
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.X/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-QXLXSXPODCNPORXU
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-KDGKRHAXXXGRNXXX
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-RXNHSUXKIXXPGTVU
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j KUBE-SVC-DGQOXKXGNXWXRSGK
-A KUBE-SERVICES -m comment --comment "comment" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-XXXUVXNQMWXYSWTK -m comment --comment "comment" -j KUBE-SEP-ZXJGHRWWGEGRZTKX
-A KUBE-SVC-XXNXPPPXLHXVSKXT -m comment --comment "comment" -j KUBE-SEP-XXKCVBUXXXLZXVQG
-A KUBE-SVC-XXDXXMLSXXZZXWBX -m comment --comment "comment" -j KUBE-SEP-YXEQXVIXWLOAYPXY
-A KUBE-SVC-XAMYINUXQIQJNTKI -m comment --comment "comment" -j KUBE-SEP-DXXFXHFXXLXZOBGD
-A KUBE-SVC-XDALDXMRMYUXXXXW -m comment --comment "comment" -j KUBE-SEP-LXYLUAILFMGAZHGA
-A KUBE-SVC-XDDEXMFNHNSQHKLA -m comment --comment "comment" -j KUBE-SEP-UMGLSKFVXXUSOMTL
-A KUBE-SVC-XFEADOKDMXXTPSOI -m comment --comment "comment" -j KUBE-SEP-XRZGCKXKVSUWTXVX
-A KUBE-SVC-XIXXHOZKPQXBJDXF -m comment --comment "comment" -j KUBE-SEP-NMYESYJXHAMBXEVX
-A KUBE-SVC-XNWHOQJXZXYDXQAX -m comment --comment "comment" -j KUBE-SEP-ZHWIXXKLTIJEXKET
-A KUBE-SVC-XPCRIRAXZQCLXXQO -m comment --comment "comment" -j KUBE-SEP-DXIZHYZXQEXBXXYQ
-A KUBE-SVC-XQSGXKGXCPLFWVQC -m comment --comment "comment" -j KUBE-SEP-NIXDSDLDDNYDRCQN
-A KUBE-SVC-XUNDXXKWWLCXFMXB -m comment --comment "comment" -j KUBE-SEP-GXXGWXLLFCXLDXDF
-A KUBE-SVC-XXWITZOXXXKQZSJN -m comment --comment "comment" -j KUBE-SEP-XXBNXAGOXXXFKJYX
-A KUBE-SVC-XAXGXYZSMCBVNXJU -m comment --comment "comment" -j KUBE-SEP-AROFZVLFCAODIYUK
-A KUBE-SVC-XAXFXWWXXMYXFKAX -m comment --comment "comment" -j KUBE-SEP-AHAFXLXAGUXXQUXP
-A KUBE-SVC-XALVSUXXVMSRVJDD -m comment --comment "comment" -j KUBE-SEP-XIXRRGOIWSTKXYSX
-A KUBE-SVC-XBJXXGHYXBSXSPHX -m comment --comment "comment" -j KUBE-SEP-UDXXXXXXTOXMPXZC
-A KUBE-SVC-XCJBOXYMSYXMXDAT -m comment --comment "comment" -j KUBE-SEP-XZXRBDLLPAKXIPZD
-A KUBE-SVC-XCMEVYXYGATYYYXX -m comment --comment "comment" -j KUBE-SEP-HYQKNCVLILQXQYQY
-A KUBE-SVC-XKAXXYRMXXPFXBND -m comment --comment "comment" -j KUBE-SEP-KVZCXXXXQNMSXZZF
-A KUBE-SVC-XSPOKYTTXEMSQXOH -m comment --comment "comment" -j KUBE-SEP-CIQZXXPZXWDZXOAK
-A KUBE-SVC-XVLUXXXUGKEGEXJC -m comment --comment "comment" -j KUBE-SEP-STZMJAGTXUPYRKXT
-A KUBE-SVC-XWJXIXXNJYJDGIXB -m comment --comment "comment" -j KUBE-SEP-RXVOXXZRPXWXGXGL
-A KUBE-SVC-XXCEIMYAEEXBXXTZ -m comment --comment "comment" -j KUBE-SEP-XIXCIFAXDJVZTSRL
-A KUBE-SVC-XYINQOWIXKZRXPOX -m comment --comment "comment" -j KUBE-SEP-IXGKMHTPNPCKXDQF
-A KUBE-SVC-XXTWNWKYZXOYZLIG -m comment --comment "comment" -j KUBE-SEP-IXXLVXULXQXXYCKG
-A KUBE-SVC-XXMTJXRTCKKYXXVU -m comment --comment "comment" -j KUBE-SEP-ESSSFRXQUDXIXXQC
-A KUBE-SVC-XEXXYJTQXQXQNGZV -m comment --comment "comment" -j KUBE-SEP-XHJPKSWXXNYXZXXC
-A KUBE-SVC-XJXIWXXNXHJBRVXX -m comment --comment "comment" -j KUBE-SEP-QJZXXNXDTXZXGYWC
-A KUBE-SVC-XJDXXICUXVGNQXQD -m comment --comment "comment" -j KUBE-SEP-DBMRXUMTBXVRSJXD
-A KUBE-SVC-XSZCKCPQDKNXDFNS -m comment --comment "comment" -j KUBE-SEP-UCXJXFLUUUMRJJYA
-A KUBE-SVC-XYOTXXXMXXXTHYYW -m comment --comment "comment" -j KUBE-SEP-KRXXMFEWCAXXLXXF
-A KUBE-SVC-XXXZXJXKCHUXXMXN -m comment --comment "comment" -j KUBE-SEP-ZAMXLSXVYXXBRUKU
-A KUBE-SVC-XOWGKPDATKXHHPYT -m comment --comment "comment" -j KUBE-SEP-ALFFLJPXKDVXNTNX
-A KUBE-SVC-XPAMRSXFYTJJJSYX -m comment --comment "comment" -j KUBE-SEP-ZFJYNOXNYDZYNUXH
-A KUBE-SVC-XRJIKSAYLIBAEXTX -m comment --comment "comment" -j KUBE-SEP-XAXMWFMJJXXXGWJU
-A KUBE-SVC-XTTMRMIXNXXOMXZK -m comment --comment "comment" -j KUBE-SEP-XBIXEZSSCVBXOTPG
-A KUBE-SVC-XXOCCNLMHHXQRKSY -m comment --comment "comment" -j KUBE-SEP-KLXTPYGHNGBNNDXC
-A KUBE-SVC-XAKOWCXINCDVIIMX -m comment --comment "comment" -j KUBE-SEP-FWTRXOFKSAINOBXB
-A KUBE-SVC-XFURLXOTYJGXIXGM -m comment --comment "comment" -j KUBE-SEP-YPSRQIRBZPXSMDAW
-A KUBE-SVC-XHZXWFZUXQFDPKQM -m comment --comment "comment" -j KUBE-SEP-VIAHEXHEXXDPVFUX
-A KUBE-SVC-XOQXXQBKMUVXZZHX -m comment --comment "comment" -j KUBE-SEP-IOXBCMXXSLXSAJXO
-A KUBE-SVC-XQXJSKEMIHWVHPEU -m comment --comment "comment" -j KUBE-SEP-BHRJDNOQXXZKRTUC
-A KUBE-SVC-XRKGQQLXKBIGCWPQ -m comment --comment "comment" -j KUBE-SEP-MXMIWXXDLEUVOERO
-A KUBE-SVC-XWIAXGLGISJXPSYX -m comment --comment "comment" -j KUBE-SEP-YXXSVELPXQSJXFEB
-A KUBE-SVC-XXXKLGXMSXTLAAXH -m comment --comment "comment" -j KUBE-SEP-XPXNRZRYBXXBWDMI
-A KUBE-SVC-XXXZXGUBIBKCXMVX -m comment --comment "comment" -j KUBE-SEP-SVAQIAHXEXXVXNXP
-A KUBE-SVC-XXSMIBXFSYXEJIXR -m comment --comment "comment" -j KUBE-SEP-ESSREXJUXFEYEUTH
-A KUBE-SVC-XANHUZMJHMXAUHCX -m comment --comment "comment" -j KUBE-SEP-KKJKRURKHXHTIRRW
-A KUBE-SVC-XDBXKXGSUZLPGKXN -m comment --comment "comment" -j KUBE-SEP-XAHRYQUUXYXIDXSI
-A KUBE-SVC-XIIGIURZARRLGVAQ -m comment --comment "comment" -j KUBE-SEP-XGQXNUXUVVREFJGE
-A KUBE-SVC-XJXGXXXKXVPPNYDX -m comment --comment "comment" -j KUBE-SEP-GXXRIGDZOHBSSXPX
-A KUBE-SVC-XKQXCLOXSVHXESIC -m comment --comment "comment" -j KUBE-SEP-GCUUDEUAQBSJXNNX
-A KUBE-SVC-XKUBSMFPMNCCWXAG -m comment --comment "comment" -j KUBE-SEP-PXEIHHMPWXRIOTXZ
-A KUBE-SVC-XMXNYMRVVMJIUWGB -m comment --comment "comment" -j KUBE-SEP-QECIAEFZDXHXIRNQ
-A KUBE-SVC-XNXCDXLNPCFXXGLT -m comment --comment "comment" -j KUBE-SEP-XWPBVGXXTDFXFYDG
-A KUBE-SVC-XOXRXBPGZXTINJQJ -m comment --comment "comment" -j KUBE-SEP-VKLMPMSXDXBEISCW
-A KUBE-SVC-XPEXXFXXDMVDXEGX -m comment --comment "comment" -j KUBE-SEP-XXOOEPTHROQWXXZB
-A KUBE-SVC-AXGHWLTXXIMXTKXK -m comment --comment "comment" -j KUBE-SEP-GXXZXMQXOXAOJXBS
-A KUBE-SVC-ACGXJWXAEQISTIOS -m comment --comment "comment" -j KUBE-SEP-EXFOQXDFXSZXZIDW
-A KUBE-SVC-AGQEAXDEKHQGCFOU -m comment --comment "comment" -j KUBE-SEP-XUCFXAHUXAXXXALA
-A KUBE-SVC-AHXXZIIRSEPXPIKX -m comment --comment "comment" -j KUBE-SEP-GIDXXMFAPLTJTTVV
-A KUBE-SVC-AICNZLUXHMXNXBXV -m comment --comment "comment" -j KUBE-SEP-XXJNSHVLAXXBZXYX
-A KUBE-SVC-ALUGOXTPGIKZDKZU -m comment --comment "comment" -j KUBE-SEP-XKNYFWSSIXNVVCPB
-A KUBE-SVC-ARZRXWKUZUHPZXPH -m comment --comment "comment" -j KUBE-SEP-XRHHCDXTPWEGIXLX
-A KUBE-SVC-ASJYXHXTFCQVNXQF -m comment --comment "comment" -j KUBE-SEP-AEVCABFEIEGRXTEK
-A KUBE-SVC-AXDKZXSCYUKMDXYX -m comment --comment "comment" -j KUBE-SEP-XHRCRWHRSCXVLUUG
-A KUBE-SVC-BXUDJOXXJHVJOHHE -m comment --comment "comment" -j KUBE-SEP-CXXLQHOXTXTRPOVG
-A KUBE-SVC-BBZSSVXFXYEXQTGN -m comment --comment "comment" -j KUBE-SEP-HXXRYXGSAACJKMQD
-A KUBE-SVC-BGWYTXCLUEZQFXTP -m comment --comment "comment" -j KUBE-SEP-XGTIANXWGTJQCXNE
-A KUBE-SVC-BLZBCBNXVRQXMIXZ -m comment --comment "comment" -j KUBE-SEP-VTFOGXMYBZLEXXUT
-A KUBE-SVC-BNQERAVTXGWNPVPB -m comment --comment "comment" -j KUBE-SEP-UBYTXZMXSEHXXWXC
-A KUBE-SVC-BPBBFYXJUBAEXDND -m comment --comment "comment" -j KUBE-SEP-XXRXZAXCZYXDZMMA
-A KUBE-SVC-BUGAXNWNXDNZXNOO -m comment --comment "comment" -j KUBE-SEP-ZEXMXXGXLXXAXQJF
-A KUBE-SVC-BXORXHXWNXZBSYXG -m comment --comment "comment" -j KUBE-SEP-FCNMQLICDXZXHGPS
-A KUBE-SVC-CXNJQJHXAUHNICZL -m comment --comment "comment" -j KUBE-SEP-OIYKXYEXNOXFYXDX
-A KUBE-SVC-CFQYBBCIXXHBFTMC -m comment --comment "comment" -j KUBE-SEP-NZIXOBCPMZOIAZFA
-A KUBE-SVC-CHRTKFCEFVMLLAOQ -m comment --comment "comment" -j KUBE-SEP-EWKXSTBCSZXXHBYH
-A KUBE-SVC-CLDVMJSPGLBDHHNW -m comment --comment "comment" -j KUBE-SEP-HSFDSOYCXLBMRHXT
-A KUBE-SVC-CTTFXKEXXTXKBSOB -m comment --comment "comment" -j KUBE-SEP-FMBHPSBNXPJHFQMN
-A KUBE-SVC-CUCLTQZFSLSSNMAB -m comment --comment "comment" -j KUBE-SEP-XYSFKYVNDXOUESZX
-A KUBE-SVC-CUHLIXXZGUXKWGUQ -m comment --comment "comment" -j KUBE-SEP-MQEWOJWPHPVOHDNX
-A KUBE-SVC-CXSBXXRXVXFGGXDH -m comment --comment "comment" -j KUBE-SEP-VSPXRXBFDYRLZIHC
-A KUBE-SVC-DXLKXPCZKQXDXHQG -m comment --comment "comment" -j KUBE-SEP-YBIJXQXYCNAAFMSX
-A KUBE-SVC-DBDXFMFHXXLBJXXC -m comment --comment "comment" -j KUBE-SEP-XVRXIRHFHURMXKXB
-A KUBE-SVC-DECQCLSHXXHXVFUM -m comment --comment "comment" -j KUBE-SEP-XXLXWSLUXFXTXSXT
-A KUBE-SVC-DGQOXKXGNXWXRSGK -m comment --comment "comment" -j KUBE-SEP-XBIBKZIWRXQFUXIU
-A KUBE-SVC-DMAQJBLZXCXGOIXJ -m comment --comment "comment" -j KUBE-SEP-JOBNTVXNLXDAWAXX
-A KUBE-SVC-DPYMXXXMXQBXZQBY -m comment --comment "comment" -j KUBE-SEP-UTXOXFNFABWHVGCX
-A KUBE-SVC-DRVTNYINXDICIJLV -m comment --comment "comment" -j KUBE-SEP-FFLSSXXXCURPOXJJ
-A KUBE-SVC-EXXFEJXTPSMZKHXS -m comment --comment "comment" -j KUBE-SEP-XIAXOXIVJMCHKMXX
-A KUBE-SVC-EXXAWQCXKMJSLTWG -m comment --comment "comment" -j KUBE-SEP-HGIVSFRTXXAXNHXX
-A KUBE-SVC-EXXXWXUXJXIJYRCX -m comment --comment "comment" -j KUBE-SEP-IUXCMFIVXXTXAINK
-A KUBE-SVC-EIDXXXJUJXEXDLWF -m comment --comment "comment" -j KUBE-SEP-XJBHGVMWXXSXXJIA
-A KUBE-SVC-ENDXXMLXICOZCDBN -m comment --comment "comment" -j KUBE-SEP-NXGXLAEZDXDQCTXJ
-A KUBE-SVC-ETKFQRWWWGDOOZXX -m comment --comment "comment" -j KUBE-SEP-EQOLXXXHXOXKXSHX
-A KUBE-SVC-EUXORBTXHFRIXGXJ -m comment --comment "comment" -j KUBE-SEP-MNCXKIDAGVJPJJKO
-A KUBE-SVC-EUFAGUTGVTRLBVXA -m comment --comment "comment" -j KUBE-SEP-XXPJZHXOJGZJFDYA
-A KUBE-SVC-EVNLDBXXEEALUPVJ -m comment --comment "comment" -j KUBE-SEP-LHHXGNGPUKWXONLW
-A KUBE-SVC-EXLCBKKQPWVSSXCX -m comment --comment "comment" -j KUBE-SEP-XOXOKVYPYXKTUSSB
-A KUBE-SVC-FXFACXTYJUXXBTXW -m comment --comment "comment" -j KUBE-SEP-DXJVGTXXONKLONXZ
-A KUBE-SVC-FXPOFXXXTGXCWNMX -m comment --comment "comment" -j KUBE-SEP-TPXFEZTUIKWENXLX
-A KUBE-SVC-FADXWXPXVRGFGXNK -m comment --comment "comment" -j KUBE-SEP-AKXTJHXIXXXYKOXE
-A KUBE-SVC-FBXIWUMANQGXXQXR -m comment --comment "comment" -j KUBE-SEP-ERXMCXSJXRQRXOXE
-A KUBE-SVC-FCXCZYXPWZFJWQPS -m comment --comment "comment" -j KUBE-SEP-MRXXFQXXXFCPRDEF
-A KUBE-SVC-FCNNXXBSOGILXJZT -m comment --comment "comment" -j KUBE-SEP-XXDIJNCXZNXRXCDK
-A KUBE-SVC-FGSYZTMXXXAPVYYV -m comment --comment "comment" -j KUBE-SEP-LZXJDAONXXXBPHXB
-A KUBE-SVC-FRPIDXWEEJQFXSRQ -m comment --comment "comment" -j KUBE-SEP-CXXXSQXWXXXJBIYX
-A KUBE-SVC-FSXXDXJTUHHJSOGX -m comment --comment "comment" -j KUBE-SEP-AJFMIXBSJXVEXXXT
-A KUBE-SVC-FSOXNXXGBKQZCBXO -m comment --comment "comment" -j KUBE-SEP-MAXXHNMXYXTIXXXH
-A KUBE-SVC-FTOXJRXWGGBXAXXY -m comment --comment "comment" -j KUBE-SEP-XINDJEXBGAFNKMDK
-A KUBE-SVC-GXUXWSWNZPMXBWIL -m comment --comment "comment" -j KUBE-SEP-QOFMVTGKEPYMXNJX
-A KUBE-SVC-GLMGVCLJGXAMVQKL -m comment --comment "comment" -j KUBE-SEP-GAOPRXWQBCCEKBON
-A KUBE-SVC-GMVDXGTFUDJCIXNN -m comment --comment "comment" -j KUBE-SEP-VXXXDZZJXXYXCAXV
-A KUBE-SVC-GSVDZGXEGXJXLCLS -m comment --comment "comment" -j KUBE-SEP-XGWLAZBOXUKXFXKX
-A KUBE-SVC-GZXSVSXXJQUIFLBW -m comment --comment "comment" -j KUBE-SEP-YMVXFQWJTNYIXUQX
-A KUBE-SVC-HXEAXJVCUXLHTQHX -m comment --comment "comment" -j KUBE-SEP-KEXDLPXIMJFBYEQX
-A KUBE-SVC-HXWWXMSYXBPPXBBI -m comment --comment "comment" -j KUBE-SEP-GHGUXUSZFMXIMXIB
-A KUBE-SVC-HXBJECVXSXFAAZUM -m comment --comment "comment" -j KUBE-SEP-BLMPNFXAIDYJSKXO
-A KUBE-SVC-HXOBXZXBOZXXHKXL -m comment --comment "comment" -j KUBE-SEP-RTVFTXXXSJDZUTXX
-A KUBE-SVC-HXVAEEFSXVNKZBGX -m comment --comment "comment" -j KUBE-SEP-EMSIJAKTGRDEXXGJ
-A KUBE-SVC-HXXLWLHBNHAAXXAJ -m comment --comment "comment" -j KUBE-SEP-OLXUXSFXWZAQXQXJ
-A KUBE-SVC-HCUCOIQKEBEXSKGA -m comment --comment "comment" -j KUBE-SEP-XGXYXOQECSCHYQSM
-A KUBE-SVC-HFKXCXHXRIVDXPBF -m comment --comment "comment" -j KUBE-SEP-NUJRJIJXJSISCXUL
-A KUBE-SVC-HFSWQPRTPAYXOXUD -m comment --comment "comment" -j KUBE-SEP-XXLXYXSXXXYXQXYX
-A KUBE-SVC-HIEPXXBXKFWIZAVE -m comment --comment "comment" -j KUBE-SEP-KKXXOIXXCXFYGMBH
-A KUBE-SVC-HNARAAYRXVOXMGQX -m comment --comment "comment" -j KUBE-SEP-FTSYCICMKRQYJBZH
-A KUBE-SVC-HSHATXXRTQXUIARR -m comment --comment "comment" -j KUBE-SEP-HATXEDGCLVXINEXH
-A KUBE-SVC-HTCFDVGOHITFJWIC -m comment --comment "comment" -j KUBE-SEP-XNFXZXYXWDJZKDLD
-A KUBE-SVC-HURETYBXPFFJHQXN -m comment --comment "comment" -j KUBE-SEP-AMGCZYBIAHHUXXXX
-A KUBE-SVC-HWPEQXXGXJXXXXMW -m comment --comment "comment" -j KUBE-SEP-XCFAFXXPXUXXNXXX
-A KUBE-SVC-HWVEXWXITKXWJXJF -m comment --comment "comment" -j KUBE-SEP-PQFZVDXLXXXCDDHZ
-A KUBE-SVC-HXEIXSWSWQTIQIGW -m comment --comment "comment" -j KUBE-SEP-JUJXSSMXDGDXUKCX
-A KUBE-SVC-HYSKHXNAXXFLEKEX -m comment --comment "comment" -j KUBE-SEP-CZVAWNRXDWMSIVNA
-A KUBE-SVC-IXOBPOCXDNVHLGXZ -m comment --comment "comment" -j KUBE-SEP-XHFUTFXXXOSXVVHH
-A KUBE-SVC-IXXXYOZSRXHJMCIV -m comment --comment "comment" -j KUBE-SEP-XXNGAFXPKVDCBEOE
-A KUBE-SVC-IXOMXMCANYROQEAU -m comment --comment "comment" -j KUBE-SEP-PBKBKAYRMXOXHFLQ
-A KUBE-SVC-ICPIXTXGWZNXCXWY -m comment --comment "comment" -j KUBE-SEP-TMXAZNXUSICXXWMB
-A KUBE-SVC-IKRDPXFVXXNZVXHM -m comment --comment "comment" -j KUBE-SEP-PPZLJYXXUZXKKXSS
-A KUBE-SVC-IMJRWTGXWIXGXIXM -m comment --comment "comment" -j KUBE-SEP-XBGYCPBSKXXKXOOJ
-A KUBE-SVC-IWSXXLFASGNXTXXH -m comment --comment "comment" -j KUBE-SEP-NLSXKXNQEAZULNJX
-A KUBE-SVC-JXAYLUQEXBLBWONN -m comment --comment "comment" -j KUBE-SEP-TXXYJFUREXXDLLTG
-A KUBE-SVC-JXJOXPJFEXGEDUET -m comment --comment "comment" -j KUBE-SEP-QXMKRQTRXXYGNDSU
-A KUBE-SVC-JXOPJBELBBRTOFNE -m comment --comment "comment" -j KUBE-SEP-AKQXXGVYESGFXLFO
-A KUBE-SVC-JXZRMVVZTIAJBHNQ -m comment --comment "comment" -j KUBE-SEP-RBXHXYAUHDGHENDL
-A KUBE-SVC-JXYUGXLXAXXAPFWQ -m comment --comment "comment" -j KUBE-SEP-EVXXCFGMAOUPYJRU
-A KUBE-SVC-JEVUQIBXQIOXIHLG -m comment --comment "comment" -j KUBE-SEP-XTJZYLXVVPCOXLWH
-A KUBE-SVC-JFSRBXUHXPJFKKPC -m comment --comment "comment" -j KUBE-SEP-OXNXGYBXXNHVTWXG
-A KUBE-SVC-JMQXOSEYEXBLMEXD -m comment --comment "comment" -j KUBE-SEP-YXXUHBBVXRPKUPPY
-A KUBE-SVC-JQTIXPXXCHQAQVMX -m comment --comment "comment" -j KUBE-SEP-MMFRTYIXXTIXBJQU
-A KUBE-SVC-JRPXRIORNTHOXXXD -m comment --comment "comment" -j KUBE-SEP-MXDEFXGZSWCWDDXM
-A KUBE-SVC-KDGKRHAXXXGRNXXX -m comment --comment "comment" -j KUBE-SEP-XUJHIJKZBXHGAKXS
-A KUBE-SVC-KIXYXKVWPIWNUXQP -m comment --comment "comment" -j KUBE-SEP-GQXKJUBTXXZXSPQV
-A KUBE-SVC-KLXNVPJORZNUZIXX -m comment --comment "comment" -j KUBE-SEP-XBGXPXYXLKZVXXIA
-A KUBE-SVC-KQUCXEVGZPBXQZXH -m comment --comment "comment" -j KUBE-SEP-XRCMSDHCNWFSGCZG
-A KUBE-SVC-KVQKXXPKTGXFEVSF -m comment --comment "comment" -j KUBE-SEP-FOCKHSOXXYZXGXMQ
-A KUBE-SVC-LXPIFDXJEXHYXDEV -m comment --comment "comment" -j KUBE-SEP-XBKXXGQQCWXLAXYD
-A KUBE-SVC-LXDUARETTKXEOQOD -m comment --comment "comment" -j KUBE-SEP-EXDXURVOXUWSRDXD
-A KUBE-SVC-LXIIXLJXFBXEJXMS -m comment --comment "comment" -j KUBE-SEP-MXBXXCCFXMSUDXOB
-A KUBE-SVC-LCQXWTXWEMZGXZXM -m comment --comment "comment" -j KUBE-SEP-REXPQNMGXXAASXPS
-A KUBE-SVC-LCTOAEXJXXAPQHVZ -m comment --comment "comment" -j KUBE-SEP-BXXBQRXTLFXNXDJX
-A KUBE-SVC-LHNNXYBZAAJXXCXN -m comment --comment "comment" -j KUBE-SEP-XHXSXRXVRXXXFOUP
-A KUBE-SVC-LKXQVTQXPMBUEXCU -m comment --comment "comment" -j KUBE-SEP-XXPIAXSAIFUHQGND
-A KUBE-SVC-LSJKXKPVQBXMANAL -m comment --comment "comment" -j KUBE-SEP-TXWRXXLDXUJMPSSU
-A KUBE-SVC-LUFMPUTDZOXTCBXV -m comment --comment "comment" -j KUBE-SEP-PIEADXBQPJGHEXIW
-A KUBE-SVC-LVXJYIMGFNPIFZQB -m comment --comment "comment" -j KUBE-SEP-TXHXCMUJUGEXIIVX
-A KUBE-SVC-MXXFXGXXQOJCNQXG -m comment --comment "comment" -j KUBE-SEP-HXJCXHXXOTYFZRXX
-A KUBE-SVC-MBGYWZNVXKQSNXQP -m comment --comment "comment" -j KUBE-SEP-XPNHQHILSTXODFPD
-A KUBE-SVC-MDXQDSJFMWZGNAKX -m comment --comment "comment" -j KUBE-SEP-FNQXHMYFXFGLPUXC
-A KUBE-SVC-MEUIGUHXHTOXLTXA -m comment --comment "comment" -j KUBE-SEP-HYEKGUMHLPORRXFX
-A KUBE-SVC-MYQLZXIXODHNIXXY -m comment --comment "comment" -j KUBE-SEP-AMJEXFSWNSJLDDAC
-A KUBE-SVC-NXGUSAELHXURYEXX -m comment --comment "comment" -j KUBE-SEP-XSXIXXXPHIHIPXLE
-A KUBE-SVC-NJLZPFFFNZXPXGAH -m comment --comment "comment" -j KUBE-SEP-LBOGUSDMBAJJDOXK
-A KUBE-SVC-NORIXMEBTFXKXVVN -m comment --comment "comment" -j KUBE-SEP-XMIJSFZXSOXERHCE
-A KUBE-SVC-NPXXXMXPTMTKRNXY -m comment --comment "comment" -j KUBE-SEP-VVNXTXOFJOXIVXXY
-A KUBE-SVC-NQZPYGXSXZXJXJAV -m comment --comment "comment" -j KUBE-SEP-JRZSWCXXXXDAXQCH
-A KUBE-SVC-NRAVLGXVZBYXOXDP -m comment --comment "comment" -j KUBE-SEP-LKAXZUEIEXHGXEXX
-A KUBE-SVC-NWUXXNQRGOFTYRVA -m comment --comment "comment" -j KUBE-SEP-XPYSZLKAPUWWUNKD
-A KUBE-SVC-NXBZKXFOOJLYMAKW -m comment --comment "comment" -j KUBE-SEP-ZZXRXUIHXXXYXIRL
-A KUBE-SVC-NZPXYXZEJXVXXWTL -m comment --comment "comment" -j KUBE-SEP-CXXZIBLLCLRNKDXX
-A KUBE-SVC-OXBXWXXSCGGKQGMG -m comment --comment "comment" -j KUBE-SEP-KXCBJXXTNTJXTEIX
-A KUBE-SVC-OXRXDXUXAANXTOXX -m comment --comment "comment" -j KUBE-SEP-OPJXLKWNWPQXVXXH
-A KUBE-SVC-OXUEESQXMXZCXTLV -m comment --comment "comment" -j KUBE-SEP-XXNKLXKTWBXXGGRM
-A KUBE-SVC-OFHLXJXNWGCJNZEY -m comment --comment "comment" -j KUBE-SEP-RXZZWVXAXAIFMFXX
-A KUBE-SVC-OHKMZXJVXYXGRXGZ -m comment --comment "comment" -j KUBE-SEP-ZPXIDVIKXOVXXCIA
-A KUBE-SVC-OIXXXIXGAYANIUNY -m comment --comment "comment" -j KUBE-SEP-XXIXWXTXXZWQYCUF
-A KUBE-SVC-OMKOKTXOFHUGWLJN -m comment --comment "comment" -j KUBE-SEP-XCJATFMBWNQXFXNS
-A KUBE-SVC-ONMXRXOXHRXCOXGH -m comment --comment "comment" -j KUBE-SEP-LYZJOBIRJOXAWBOO
-A KUBE-SVC-OXGEUBSAOXPWSXTF -m comment --comment "comment" -j KUBE-SEP-FTDXXXXXGPTBZRVX
-A KUBE-SVC-OZGQFFEXIIXHZXBJ -m comment --comment "comment" -j KUBE-SEP-XXODXIDXOHNUJZWZ
-A KUBE-SVC-PXRUSNTZUGGXOFXX -m comment --comment "comment" -j KUBE-SEP-GPUXOXSQSFYNYWXX
-A KUBE-SVC-PXLBBYXEXXXDSXHM -m comment --comment "comment" -j KUBE-SEP-UXXPEXMZJDJFNHXX
-A KUBE-SVC-PBMZKPRXSXPWXNQG -m comment --comment "comment" -j KUBE-SEP-LQAALXZAKRVXBRXA
-A KUBE-SVC-PDXYXXUOWQMEIXXE -m comment --comment "comment" -j KUBE-SEP-EDASXVXXMMUKUEXX
-A KUBE-SVC-PEWTBIVHUXXSOFYV -m comment --comment "comment" -j KUBE-SEP-XXXXILOCXGNBWYKX
-A KUBE-SVC-PNRPNORAEXDKHKTY -m comment --comment "comment" -j KUBE-SEP-DUUXNYOZROGKKSIM
-A KUBE-SVC-PUXGQXWTKXVEZXWE -m comment --comment "comment" -j KUBE-SEP-AJXXRTXONUXGZJXT
-A KUBE-SVC-QXLXSXPODCNPORXU -m comment --comment "comment" -j KUBE-SEP-SUADNDUYKXMQLALM
-A KUBE-SVC-QXPLXPGLXRCUPXWG -m comment --comment "comment" -j KUBE-SEP-SXQXAJMXYMXXXTEX
-A KUBE-SVC-QBZOPXKIXXBWRUXB -m comment --comment "comment" -j KUBE-SEP-FXXVXQJVXXXZMMEX
-A KUBE-SVC-QFDJBTXXHXVFXQXB -m comment --comment "comment" -j KUBE-SEP-SLXJBTMZUNNFRHKS
-A KUBE-SVC-QGQUXGAQXHKLJXGY -m comment --comment "comment" -j KUBE-SEP-TOQEXXXPOKTVCDEA
-A KUBE-SVC-QISOODXAQKJMISXT -m comment --comment "comment" -j KUBE-SEP-PFXQQXCBPXFCJOXS
-A KUBE-SVC-QKVXBXQYQSSXXOFM -m comment --comment "comment" -j KUBE-SEP-XVREBNCXQXVXUHUW
-A KUBE-SVC-QRTPUHKXXCZKXDCR -m comment --comment "comment" -j KUBE-SEP-XXCSXBXVBPXHXHVE
-A KUBE-SVC-QWXBEWKEUXPNWITP -m comment --comment "comment" -j KUBE-SEP-CEOAQHQEICXMXXHD
-A KUBE-SVC-RXXEXKNXHXSXOLXX -m comment --comment "comment" -j KUBE-SEP-YUXKYZOAIIOLXPGJ
-A KUBE-SVC-RXXJODPXIRXWGTBE -m comment --comment "comment" -j KUBE-SEP-APXHMALBXDXHLZCX
-A KUBE-SVC-RXNHSUXKIXXPGTVU -m comment --comment "comment" -j KUBE-SEP-XCFYDDJVFZXSQJXU
-A KUBE-SVC-RBRSHHXAXXQKWEBR -m comment --comment "comment" -j KUBE-SEP-RRJHMMXBCKNFAXGU
-A KUBE-SVC-RCXXHARMXHQXWYCY -m comment --comment "comment" -j KUBE-SEP-XLTTNSWMTXQCZQTJ
-A KUBE-SVC-RCXMYQNMDOPXVQBX -m comment --comment "comment" -j KUBE-SEP-XPUCLUJXGNJMQURT
-A KUBE-SVC-RGOXTPKGXYZXXXVJ -m comment --comment "comment" -j KUBE-SEP-OSKWUDBAZCYEXMXX
-A KUBE-SVC-RNXERQXBEXZDIJQX -m comment --comment "comment" -j KUBE-SEP-XHRAZUVCRXGRYEXX
-A KUBE-SVC-RRRTXYQBWWKJDBGX -m comment --comment "comment" -j KUBE-SEP-JXZYVTIQGTWNQGWA
-A KUBE-SVC-RVQBCAXUXVCVZORX -m comment --comment "comment" -j KUBE-SEP-ESYUEXZIXMXXOZUX
-A KUBE-SVC-SXGNXXRFSNZJQGBX -m comment --comment "comment" -j KUBE-SEP-XNAAXXOPTCMXKXYM
-A KUBE-SVC-SFGNJEIRRIUXMFRW -m comment --comment "comment" -j KUBE-SEP-ORLFLCRNYFTCFDXT
-A KUBE-SVC-SMXXRRPEUKMAAQXX -m comment --comment "comment" -j KUBE-SEP-LHBORJVHQFXXDEHT
-A KUBE-SVC-SXCPBCXYGRYYSULL -m comment --comment "comment" -j KUBE-SEP-USVXTXXIKXLBNCPB
-A KUBE-SVC-TXGRNXIHRFVEYXPT -m comment --comment "comment" -j KUBE-SEP-ZHDHXBXJLGXDXRXD
-A KUBE-SVC-TXXGQXCJOMVOZBBX -m comment --comment "comment" -j KUBE-SEP-DNMECZZXLFQXXIMD
-A KUBE-SVC-TXIJAXWADCNXVDYU -m comment --comment "comment" -j KUBE-SEP-RSXTXXXEJMXPXOXE
-A KUBE-SVC-TXXAGXXTIOKPXUXV -m comment --comment "comment" -j KUBE-SEP-XAXPXGVMAZEXPTBX
-A KUBE-SVC-TBXCYTQSHMMKCABN -m comment --comment "comment" -j KUBE-SEP-XASJXLVYYABDOXRR
-A KUBE-SVC-TIGHJYAXIGXLXQUB -m comment --comment "comment" -j KUBE-SEP-XGHZLNYDTXGPXVXZ
-A KUBE-SVC-TMSWZXXUSXHGQCEC -m comment --comment "comment" -j KUBE-SEP-IFXCFXFXYZSDGXDC
-A KUBE-SVC-UXWPUXBMDXIXXVXR -m comment --comment "comment" -j KUBE-SEP-XUNLVRURGXZOTLXJ
-A KUBE-SVC-UXLGJYXXAOEXAXCN -m comment --comment "comment" -j KUBE-SEP-XKNFNNDXSFQKMGDR
-A KUBE-SVC-UXVXGQPGYBXCXHBA -m comment --comment "comment" -j KUBE-SEP-VJXNTJXTXJRNWPUD
-A KUBE-SVC-UCKRSXVFXLWRBQXX -m comment --comment "comment" -j KUBE-SEP-ZHXAUOTEHBQMELPQ
-A KUBE-SVC-UIPBDKQXOEIOTXXU -m comment --comment "comment" -j KUBE-SEP-XRPQZGHSFAJEXXIP
-A KUBE-SVC-UMDLOONOEFMGBPAY -m comment --comment "comment" -j KUBE-SEP-XXDXHIGQIZHPOBEL
-A KUBE-SVC-URRVHCUSXHVBWZEX -m comment --comment "comment" -j KUBE-SEP-DMNYWOPVWRDKXOTX
-A KUBE-SVC-VXDLMYAXKWVXUTXX -m comment --comment "comment" -j KUBE-SEP-BRDXGGFXXYBFPSJT
-A KUBE-SVC-VFADXEYXZKNUGSEX -m comment --comment "comment" -j KUBE-SEP-EXXNBXDNWCWGTXQT
-A KUBE-SVC-VKLGXKCXGMVXWUEB -m comment --comment "comment" -j KUBE-SEP-UCETASOSIXJDXZXH
-A KUBE-SVC-VMMVURAXDBXXXXJN -m comment --comment "comment" -j KUBE-SEP-AEXDRVALKCXYXGMW
-A KUBE-SVC-VPVJXGTKLPDXGBXM -m comment --comment "comment" -j KUBE-SEP-OXRXPXXDOHCVUAEM
-A KUBE-SVC-VSXXRGEHXXNYHXFJ -m comment --comment "comment" -j KUBE-SEP-FEDZSXKERFEDKMDY
-A KUBE-SVC-VYXOQXXZWIXUPHVS -m comment --comment "comment" -j KUBE-SEP-VZSUSPSQYXXCXCXP
-A KUBE-SVC-WXXITJBXZTXLRQTL -m comment --comment "comment" -j KUBE-SEP-CBXTTPIDEXKXXNEV
-A KUBE-SVC-WXTSXSXXXJTYQTXC -m comment --comment "comment" -j KUBE-SEP-FDFXZYXLXZOXUNRB
-A KUBE-SVC-WXUIXXNJXDGCJDNE -m comment --comment "comment" -j KUBE-SEP-OPAVXQXNDINUXESF
-A KUBE-SVC-WXIKJOVUDXGZSFCY -m comment --comment "comment" -j KUBE-SEP-PDOBQZHDNANTXLZX
-A KUBE-SVC-WGZMXNSXKCTBXXHX -m comment --comment "comment" -j KUBE-SEP-XBFJMUPXHAZLXXXE
-A KUBE-SVC-WJXRTRAQDWVXWXUW -m comment --comment "comment" -j KUBE-SEP-GKAYASHWKXGHUSFX
-A KUBE-SVC-WNTRXXZETDXWAXXG -m comment --comment "comment" -j KUBE-SEP-IZPCXCXQXRXMHVLX
-A KUBE-SVC-WQNUPYEZVKLVRPCZ -m comment --comment "comment" -j KUBE-SEP-XIQWDXXBCXVOXZXB
-A KUBE-SVC-WUGXWRBAXOXANDDK -m comment --comment "comment" -j KUBE-SEP-CDBMXXGNMILHQLXV
-A KUBE-SVC-WXXGPAQXELITGJRJ -m comment --comment "comment" -j KUBE-SEP-XWDLVELPFVUPQTXV
-A KUBE-SVC-WYXLXLXXSMXXFHNL -m comment --comment "comment" -j KUBE-SEP-WNNUSXXCXBXESMXA
-A KUBE-SVC-WZXXKXXTYKVIQOXN -m comment --comment "comment" -j KUBE-SEP-JFEXFKXXXMXSDMVI
-A KUBE-SVC-XXNQGLSXEOXQXDXF -m comment --comment "comment" -j KUBE-SEP-CPCDSSDEDRQBVJCV
-A KUBE-SVC-XXHWYXEXXXMGKHYU -m comment --comment "comment" -j KUBE-SEP-XZXXWIXGPGHXFHSX
-A KUBE-SVC-XXPJXGBBHJKAFHFG -m comment --comment "comment" -j KUBE-SEP-TXMKXKFHXZXXWIXK
-A KUBE-SVC-XCQXYGSPTQXPXXXN -m comment --comment "comment" -j KUBE-SEP-IDXXIXVVUPCWOZBX
-A KUBE-SVC-XCXNXMCRNAULYXWN -m comment --comment "comment" -j KUBE-SEP-XSJLKSXTXXCQXCXX
-A KUBE-SVC-XDXVZTEZJCXVWGTQ -m comment --comment "comment" -j KUBE-SEP-ATXKFPRQXCTRNXSE
-A KUBE-SVC-XHXPAAZVXBMVXALJ -m comment --comment "comment" -j KUBE-SEP-TVLLXEGQZHXXJBKW
-A KUBE-SVC-XHTBCBWAAQTDXIXJ -m comment --comment "comment" -j KUBE-SEP-XXZBGBXOJKUEVCXO
-A KUBE-SVC-XLYZVSOKYJUXXOIP -m comment --comment "comment" -j KUBE-SEP-URUQSXXXPEXDMXTF
-A KUBE-SVC-XOWEHFXSGCMCXBXX -m comment --comment "comment" -j KUBE-SEP-XXIONLXMLGFZOZFB
-A KUBE-SVC-XQXGSAEXPJFQYRJQ -m comment --comment "comment" -j KUBE-SEP-DKXMXFNXTBESXDLE
-A KUBE-SVC-XRAYXJPZXMFQXSRX -m comment --comment "comment" -j KUBE-SEP-VKXPCXTMFURRPXXX
-A KUBE-SVC-XVKLXNZPBXXLXZDU -m comment --comment "comment" -j KUBE-SEP-UONYXWXXRHJALCSS
-A KUBE-SVC-XYIXQRQCWSXADXLG -m comment --comment "comment" -j KUBE-SEP-JXSXTLBYGWDHBQXS
-A KUBE-SVC-YXYBMOOOXHKDLTCX -m comment --comment "comment" -j KUBE-SEP-IMXOZNUHPVKNJOCG
-A KUBE-SVC-YDRXHBXMXDXLOXVE -m comment --comment "comment" -j KUBE-SEP-XIAZXNFOJXDGWPXX
-A KUBE-SVC-YDXANXNUOSFKKXKW -m comment --comment "comment" -j KUBE-SEP-GACXVHRXIXXNCYSG
-A KUBE-SVC-YFXARXCQXCFBDEKB -m comment --comment "comment" -j KUBE-SEP-BYGNXIVTCZNXXXNV
-A KUBE-SVC-YNMLSRXXMXCNMJXT -m comment --comment "comment" -j KUBE-SEP-ZBQPUGNYSXXHSKOO
-A KUBE-SVC-YUGSJKVOJTGRXBOL -m comment --comment "comment" -j KUBE-SEP-IXVXPLSXZBBVDACE
-A KUBE-SVC-YVUMXXXHOGFYBHPU -m comment --comment "comment" -j KUBE-SEP-NXXHEXHAJBWZANYE
-A KUBE-SVC-YYGHYHVRZXNERXXX -m comment --comment "comment" -j KUBE-SEP-PXWUXNTCIGZXONUA
-A KUBE-SVC-ZXXXVMHUFXXRRHZA -m comment --comment "comment" -j KUBE-SEP-QBJAQUBXHUBBUBXT
-A KUBE-SVC-ZXCEXHGUXXFBIXVI -m comment --comment "comment" -j KUBE-SEP-JPXKXXNXYPXQTZXE
-A KUBE-SVC-ZBDQRWJXJGOZVZJL -m comment --comment "comment" -j KUBE-SEP-XIIVCPFXKMCXHWBX
-A KUBE-SVC-ZCXXTYSUJJEZXXXF -m comment --comment "comment" -j KUBE-SEP-VKDULWHHBYJKYKZX
-A KUBE-SVC-ZGZMOXKMXFCGXXKX -m comment --comment "comment" -j KUBE-SEP-VNSXSXXRGGYEHXXY
-A KUBE-SVC-ZIFWATGFWIJZQKXI -m comment --comment "comment" -j KUBE-SEP-WWXKKXPXZISFXWYY
-A KUBE-SVC-ZLAOWJUEHQXXTZFK -m comment --comment "comment" -j KUBE-SEP-LACEFMXTWPWGIOTJ
-A KUBE-SVC-ZLINXJCXIHPXTXPX -m comment --comment "comment" -j KUBE-SEP-XGCOHXVPDXTXIZUS
-A KUBE-SVC-ZLSRFQYMOZMAZXSP -m comment --comment "comment" -j KUBE-SEP-QRXUQSXMSFDVZMYZ
-A KUBE-SVC-ZQJNYVXRXXOWSDOS -m comment --comment "comment" -j KUBE-SEP-XMOMWXXUCGJWUDWX
-A KUBE-SVC-ZSUHDTKEKQDANICV -m comment --comment "comment" -j KUBE-SEP-SZXOYSXJXXQFXXIT
-A KUBE-SVC-ZTAHJCVAMDTXDLDZ -m comment --comment "comment" -j KUBE-SEP-FQXVOLQGTXXBXUXN
-A KUBE-SVC-ZUMKTXFQRYYYIPCX -m comment --comment "comment" -j KUBE-SEP-MOXDXQXIOXBELXWX
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX
# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*filter
:INPUT DROP XX:XX
:FORWARD ACCEPT XXX:XXXX
:OUTPUT ACCEPT XXX:XXXXXX
:KUBE-FIREWALL - XX:XX
:KUBE-SERVICES - XX:XX
-A INPUT -j KUBE-FIREWALL
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m pkttype --pkt-type multicast -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A INPUT -p icmp -j ACCEPT
-A INPUT -s XX.XX.XX.XXX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XXX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XXX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -p udp -m udp --dport XXXX -j ACCEPT
-A INPUT -s XX.XX.XX.XXX/XX -j ACCEPT
-A INPUT -s XX.XXX.X.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.XXX.XX.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.XXX.XX.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.XXX.XX.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.XX.X.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.XX.XX.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.XX.XX.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.XX.XX.X/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A INPUT -s XX.X.X.X/X -p tcp -m tcp --dport X:XXXX -j ACCEPT
-A INPUT -s XX.X.X.X/X -p tcp -m tcp --dport XXXX:XXXXX -j ACCEPT
-A INPUT -s XX.X.X.X/X -p tcp -m tcp --dport XXXXX:XXXXX -j ACCEPT
-A INPUT -s XX.X.X.X/X -p tcp -m tcp --dport XXXXX:XXXXX -j ACCEPT
-A INPUT -s XX.XX.XX.XX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XX/XX -p tcp -m tcp --dport XX -j ACCEPT
-A INPUT -s XX.XX.XX.XXX/XX -p tcp -m tcp --dport XXXXX -j ACCEPT
-A INPUT -s XX.XX.XX.XXX/XX -p tcp -m tcp --dport XXXXX -j ACCEPT
-A OUTPUT -m comment --comment "comment" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "comment" -m mark --mark XxXXXX/XxXXXX -j DROP
-A KUBE-SERVICES -d XXX.XXX.XX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.X.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.XXX.XXX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.X.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d XXX.XXX.XX.XX/XX -p tcp -m comment --comment "comment" -m tcp --dport XXXX -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX
[-- Attachment #3: openstack-iptables.txt --]
[-- Type: text/plain, Size: 48414 bytes --]
# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*filter
:INPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXXX
:FORWARD ACCEPT XXXXXXXXXXX:XXXXXXXXXXXX
:OUTPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXX
:nova-api-FORWARD - XX:XX
:nova-api-INPUT - XX:XX
:nova-api-OUTPUT - XX:XX
:nova-api-local - XX:XX
:nova-filter-top - XX:XX
:nova-network-FORWARD - XX:XX
:nova-network-INPUT - XX:XX
:nova-network-OUTPUT - XX:XX
:nova-network-local - XX:XX
-A INPUT -j nova-network-INPUT
-A INPUT -j nova-api-INPUT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-network-FORWARD
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-api-OUTPUT
-A nova-api-INPUT -d XX.XX.XX.XX/XX -p tcp -m tcp --dport XXXX -j ACCEPT
-A nova-filter-top -j nova-network-local
-A nova-filter-top -j nova-api-local
-A nova-network-FORWARD -i brXXXX -j ACCEPT
-A nova-network-FORWARD -o brXXXX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p udp -m udp --dport XX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p tcp -m tcp --dport XX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p udp -m udp --dport XX -j ACCEPT
-A nova-network-INPUT -i brXXXX -p tcp -m tcp --dport XX -j ACCEPT
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX
# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*mangle
:PREROUTING ACCEPT XXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXX
:INPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXXX
:FORWARD ACCEPT XXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXX
:OUTPUT ACCEPT XXXXXXXXXXX:XXXXXXXXXXXXXX
:POSTROUTING ACCEPT XXXXXXXXXXXXXX:XXXXXXXXXXXXXXXXX
:nova-api-POSTROUTING - XX:XX
:nova-network-POSTROUTING - XX:XX
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-api-POSTROUTING
-A nova-network-POSTROUTING -o brXXXX -p udp -m udp --dport XX -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX
# Generated by iptables-save vX.X.XX on Mon Feb XX XX:XX:XX XXXX
*nat
:PREROUTING ACCEPT XXXXXXXXXXXX:XXXXXXXXXXXXXX
:INPUT ACCEPT XXXXXXXXXX:XXXXXXXXXXXX
:OUTPUT ACCEPT XXXXXXXX:XXXXXXXXXX
:POSTROUTING ACCEPT XXXXXXXXXXX:XXXXXXXXXXXX
:nova-api-OUTPUT - XX:XX
:nova-api-POSTROUTING - XX:XX
:nova-api-PREROUTING - XX:XX
:nova-api-float-snat - XX:XX
:nova-api-snat - XX:XX
:nova-network-OUTPUT - XX:XX
:nova-network-POSTROUTING - XX:XX
:nova-network-PREROUTING - XX:XX
:nova-network-float-snat - XX:XX
:nova-network-snat - XX:XX
:nova-postrouting-bottom - XX:XX
-A PREROUTING -j nova-network-PREROUTING
-A PREROUTING -j nova-api-PREROUTING
-A OUTPUT -j nova-network-OUTPUT
-A OUTPUT -j nova-api-OUTPUT
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-api-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A nova-api-snat -j nova-api-float-snat
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-OUTPUT -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XX.XX.XX.XX/XX -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XXX.XX.XXX.X/XX -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XX.X.X.X/X -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -m conntrack ! --ctstate DNAT -j ACCEPT
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.X/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-POSTROUTING -s XX.XX.XX.XXX/XX -m conntrack --ctstate DNAT -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-PREROUTING -d XXX.XXX.XXX.XXX/XX -p tcp -m tcp --dport XX -j DNAT --to-destination XX.XX.XX.XX:XXXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.X
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-PREROUTING -d XXX.XX.XXX.XXX/XX -j DNAT --to-destination XX.XX.XX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -d XX.XX.XX.X/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -d XX.XX.XX.XX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -d XX.XX.XX.XXX/XX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-float-snat -s XX.XX.XX.XXX/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-network-snat -j nova-network-float-snat
-A nova-network-snat -s XX.XX.XX.X/XX -o ethX -j SNAT --to-source XXX.XX.XXX.XXX
-A nova-postrouting-bottom -j nova-network-snat
-A nova-postrouting-bottom -j nova-api-snat
COMMIT
# Completed on Mon Feb XX XX:XX:XX XXXX
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:27 ` David Miller
2018-02-19 15:38 ` Harald Welte
@ 2018-02-19 17:41 ` Arturo Borrero Gonzalez
1 sibling, 0 replies; 73+ messages in thread
From: Arturo Borrero Gonzalez @ 2018-02-19 17:41 UTC (permalink / raw)
To: David Miller
Cc: Florian Westphal, daniel, laforge, netdev,
Netfilter Development Mailing list, alexei.starovoitov
On 19 February 2018 at 16:27, David Miller <davem@davemloft.net> wrote:
> From: Florian Westphal <fw@strlen.de>
> Date: Mon, 19 Feb 2018 16:15:55 +0100
>
>> Would you be willing to merge nftables into kernel tools directory
>> then?
>
> Did you miss the part where I explained that people explicitly disable
> NFTABLES in their kernel configs in most if not all large datacenters?
hey, you already shared several statements regarding nftables which
are not true.
Lots and lots of people are using distribution kernels, which contains
NF_TABLES config enabled (all major distros have it)
I believe people who build their own kernels are very few if you
compare with the number of people who don't (but yeah, they usually
have more money).
This may sounds as a joke, but there are *a lot* of people running
productions servers with bluetooth drivers enabled in the kconfig.
So, I can confirm that:
Lots of people and institutions are using nftables already.
Lots of people and institutions are considering transition to nftables
it from iptables.
Lots of people are running simple commodity hardware and know nothing
about smartnics or any kind of offloading
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:22 ` David Miller
@ 2018-02-19 18:05 ` Phil Sutter
2018-02-19 18:41 ` David Miller
2018-02-19 21:13 ` Florian Westphal
2018-02-20 10:44 ` Pablo Neira Ayuso
2 siblings, 1 reply; 73+ messages in thread
From: Phil Sutter @ 2018-02-19 18:05 UTC (permalink / raw)
To: David Miller
Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 12:22:26PM -0500, David Miller wrote:
> From: Phil Sutter <phil@nwl.cc>
> Date: Mon, 19 Feb 2018 18:14:11 +0100
>
> > OK, so reading between the lines you're saying that nftables project
> > has failed to provide an adequate successor to iptables?
>
> Whilst it is great that the atomic table update problem was solved, I
> think the emphasis on flexibility often at the expense of performance
> was a bad move.
I don't see a lack of performance in nftables when being compared to
iptables (as we have now). From my point of view, it's quite the
contrary: nftables did a great job in picking up iptables performance
afterthoughts (e.g. ipset) and leveraging that to the max(TM) (verdict
maps, concatenated set entries). Assuming the virtual machine design
principle isn't just marketing but sets the course for JIT ruleset
optimizations, there's some margin as well.
So from my perspective, one should say nftables increased flexibility
without sacrificing performance.
> Netfilter's chronic performance differential is why a lot of mindshare
> was lost to userspace networking technologies.
>
> Thankfully, we are gaining back a lot of that userbase with XDP and
> eBPF, thanks to the hard work of many individuals.
>
> To think that people are going to be willing to take the performance
> hit (whatever it's size) to go back to the "more flexible" nftables
> is really not a realistic expectation.
>
> And we have amassed enough interest and momentum that offloading eBPF
> in hardware on current and future hardware is happening.
>
> So I am going to direct us in directions that allow those realities to
> be taken advantage of, rather than pretending that this transition
> hasn't occurred already.
Hey, you secretly changed the topic! ;)
Yes, even with my limited experience I noticed that there is quite some
demand for even faster packet processing in Linux, mostly for rather
custom scenarios like forwarding into containers/VMs. Though my point
was about general purpose firewalling abilities in Linux, say people
securing their desktop or maintaining networks with less demands on
performance. I guess it will be a while until consumer hardware comes
with smart NICs (or they become affordable), so for those people
nftables is definitely a step forward.
Cheers, Phil
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:36 ` David Miller
2018-02-19 17:20 ` Harald Welte
2018-02-19 17:40 ` Arturo Borrero Gonzalez
@ 2018-02-19 18:06 ` Arturo Borrero Gonzalez
2018-02-19 18:43 ` David Miller
2 siblings, 1 reply; 73+ messages in thread
From: Arturo Borrero Gonzalez @ 2018-02-19 18:06 UTC (permalink / raw)
To: David Miller
Cc: laforge, daniel, netdev, Netfilter Development Mailing list,
alexei.starovoitov
On 19 February 2018 at 16:36, David Miller <davem@davemloft.net> wrote:
>
> In my opinion, any resistence to integration with eBPF and XDP will
> lead to even less adoption of netfilter as a technology.
>
> Therefore my plan is to move everything to be integrated around these
> important core technologies. For the purposes of integration, code
> coverage, performance, and the ability to juxtapose different bits of
> eBPF code into larger optimized code streams that can also be
> offloaded into hardware.
Thanks for sharing your plans. I'll share mine.
Debian already recommends using nftables rather than iptables.
Probably in the next release cycle we (Debian) will give even more
prominence to nftables by linking iptables to iptables-compat, as an
opt-in for users, so we don't break systems.
By the next-next release cycle (4+ years or so?) we will probably have
enough confidence with compat/translation tools that Debian could
fully wipe the old iptables binary to use just the nftables framework.
Same for ip6tables, arptables, ebtables.
Does this sound reasonable to you?
Yes, probably major datacenters (google? facebook?, amazon?) they
don't even care about what Debian is doing, since they are crafting
their own distro anyway.
But there are *a lot* of other people that do care about these migration plans.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:29 ` David Miller
@ 2018-02-19 18:37 ` Harald Welte
2018-02-19 18:47 ` David Miller
0 siblings, 1 reply; 73+ messages in thread
From: Harald Welte @ 2018-02-19 18:37 UTC (permalink / raw)
To: David Miller; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 12:29:08PM -0500, David Miller wrote:
> People with an Android phone in their pocket is using iptables, and
> the overhead and performance of those rules really does matter. It
> determines how long your battery life is, etc.
I am not the android expert. However, I just dumped the ruleset on my
Galaxy Tab S2 (Android 7.1.2 / LineageOS), and it was a whooping 91
rules across all tables. The longest chain interation I could spot
was 24 rules. That's not the kind of ruleset where I would expect
performance worries.
And if there was, nftables is around for quite some time and would be
much faster.
Sure, that was just one tablet, but I wonder how much Android packet
filter performance issue there are. Would be interesting to hear about
those (and on whether they benchmarked against nftables).
> > I can just as well ask how many millions of users / devices are
> > already using eBPF or XDP?
>
> Every time someone connects to a major provider, they are using it.
I was speaking of actual *users* as in indiiduals running their own
systems, companies running their own servers/datacenter. The fact that
some ISP (or its supplier) decisdes that one of my IP packets is routed
via a smartnic with XDP offloading somewhere is great, but still doesn't
turn me into a "user" of that technology. Not in my linke of thinking,
at least.
> And by in large, for system tracing and analysis eBPF is basically
> a hard requirement for people doing anything serious these days.
That's great, but misses the point. I was referring to usage in the
context of the kernel network stack. Sorry for not being explicit
enough.
Also, the entire point was about "new technologies need time to be
adopted widely". Doesn't matter which new kernel feature that is.
Sure, one data center / hosting / "cloud" provider can quickly roll out
a change in their network. But I'm referring to significant,
(Linux-)industry-wide adoption. That would first include major
distributions to include/enable/support the feature, and then people
actually building their systems/products/software on top of those.
> Please see the wonderful work by Brendan Gregg and others which has
> basically made the GPL'ing of DTrace by Oracle entirely irrelevant and
> our Linux's tracing infrastructure has become must more powerful and
> capable thanks to eBPF.
Agreed.
--
- Harald Welte <laforge@gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 18:05 ` Phil Sutter
@ 2018-02-19 18:41 ` David Miller
2018-02-19 20:41 ` Phil Sutter
0 siblings, 1 reply; 73+ messages in thread
From: David Miller @ 2018-02-19 18:41 UTC (permalink / raw)
To: phil; +Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Phil Sutter <phil@nwl.cc>
Date: Mon, 19 Feb 2018 19:05:51 +0100
> Hi David,
>
> On Mon, Feb 19, 2018 at 12:22:26PM -0500, David Miller wrote:
>> From: Phil Sutter <phil@nwl.cc>
>> Date: Mon, 19 Feb 2018 18:14:11 +0100
>>
>> > OK, so reading between the lines you're saying that nftables project
>> > has failed to provide an adequate successor to iptables?
>>
>> Whilst it is great that the atomic table update problem was solved, I
>> think the emphasis on flexibility often at the expense of performance
>> was a bad move.
>
> I don't see a lack of performance in nftables when being compared to
> iptables (as we have now). From my point of view, it's quite the
> contrary: nftables did a great job in picking up iptables performance
> afterthoughts (e.g. ipset) and leveraging that to the max(TM) (verdict
> maps, concatenated set entries). Assuming the virtual machine design
> principle isn't just marketing but sets the course for JIT ruleset
> optimizations, there's some margin as well.
>
> So from my perspective, one should say nftables increased flexibility
> without sacrificing performance.
I did not say nftables adjusted performance one way or another. It kept
it on the same order of magnitude. And this is a design decision.
> Yes, even with my limited experience I noticed that there is quite some
> demand for even faster packet processing in Linux, mostly for rather
> custom scenarios like forwarding into containers/VMs. Though my point
> was about general purpose firewalling abilities in Linux, say people
> securing their desktop or maintaining networks with less demands on
> performance.
I've always stated that low power, low end, systems are just a good
place for high performance filtering as high end ones.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 18:06 ` Arturo Borrero Gonzalez
@ 2018-02-19 18:43 ` David Miller
0 siblings, 0 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 18:43 UTC (permalink / raw)
To: arturo; +Cc: laforge, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Arturo Borrero Gonzalez <arturo@netfilter.org>
Date: Mon, 19 Feb 2018 19:06:12 +0100
> Yes, probably major datacenters (google? facebook?, amazon?) they
> don't even care about what Debian is doing, since they are crafting
> their own distro anyway. But there are *a lot* of other people that
> do care about these migration plans.
"Lots" is a big word that gets thrown around quite carelessly.
What do you imagine is the order of magnitude of cloud and big
datacenter server system deployments vs. individual servers and
whatnot?
I have to take into consideration what will really have the largest
impact on the largest number of people, and I am pretty sure I know
where that lies.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 18:37 ` Harald Welte
@ 2018-02-19 18:47 ` David Miller
0 siblings, 0 replies; 73+ messages in thread
From: David Miller @ 2018-02-19 18:47 UTC (permalink / raw)
To: laforge; +Cc: daniel, netdev, netfilter-devel, alexei.starovoitov
From: Harald Welte <laforge@gnumonks.org>
Date: Mon, 19 Feb 2018 19:37:30 +0100
> I was speaking of actual *users* as in indiiduals running their own
> systems, companies running their own servers/datacenter. The fact that
> some ISP (or its supplier) decisdes that one of my IP packets is routed
> via a smartnic with XDP offloading somewhere is great, but still doesn't
> turn me into a "user" of that technology. Not in my linke of thinking,
> at least.
I am sorry that our opinions differ.
I must consider all users of Linux both direct and indirect, to
determine impact and where resources and efforts should be allocated.
>> And by in large, for system tracing and analysis eBPF is basically
>> a hard requirement for people doing anything serious these days.
>
> That's great, but misses the point. I was referring to usage in the
> context of the kernel network stack. Sorry for not being explicit
> enough.
And that misses the point entirely.
Which is that eBPF is more than just networking, so it is missing
that this technology is not just networking specific but a kernel
wide one that is being adopted in every nook and cranny of the
kernel.
> Sure, one data center / hosting / "cloud" provider can quickly roll out
> a change in their network. But I'm referring to significant,
> (Linux-)industry-wide adoption.
Hehe, I guess whatever definition works for the position you are
trying to take.
:-)
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 18:41 ` David Miller
@ 2018-02-19 20:41 ` Phil Sutter
0 siblings, 0 replies; 73+ messages in thread
From: Phil Sutter @ 2018-02-19 20:41 UTC (permalink / raw)
To: David Miller
Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 01:41:29PM -0500, David Miller wrote:
> From: Phil Sutter <phil@nwl.cc>
> Date: Mon, 19 Feb 2018 19:05:51 +0100
>
> > On Mon, Feb 19, 2018 at 12:22:26PM -0500, David Miller wrote:
> >> From: Phil Sutter <phil@nwl.cc>
> >> Date: Mon, 19 Feb 2018 18:14:11 +0100
> >>
> >> > OK, so reading between the lines you're saying that nftables project
> >> > has failed to provide an adequate successor to iptables?
> >>
> >> Whilst it is great that the atomic table update problem was solved, I
> >> think the emphasis on flexibility often at the expense of performance
> >> was a bad move.
> >
> > I don't see a lack of performance in nftables when being compared to
> > iptables (as we have now). From my point of view, it's quite the
> > contrary: nftables did a great job in picking up iptables performance
> > afterthoughts (e.g. ipset) and leveraging that to the max(TM) (verdict
> > maps, concatenated set entries). Assuming the virtual machine design
> > principle isn't just marketing but sets the course for JIT ruleset
> > optimizations, there's some margin as well.
> >
> > So from my perspective, one should say nftables increased flexibility
> > without sacrificing performance.
>
> I did not say nftables adjusted performance one way or another. It kept
> it on the same order of magnitude. And this is a design decision.
Oh, seems I missed your point then. What subject did you have in mind
when you wrote "emphasis on flexibility often at the expense of
performance"? I thought you were talking about nftables.
> > Yes, even with my limited experience I noticed that there is quite some
> > demand for even faster packet processing in Linux, mostly for rather
> > custom scenarios like forwarding into containers/VMs. Though my point
> > was about general purpose firewalling abilities in Linux, say people
> > securing their desktop or maintaining networks with less demands on
> > performance.
>
> I've always stated that low power, low end, systems are just a good
> place for high performance filtering as high end ones.
Do you think these systems are likely to receive a NIC (or some sort of
co-processor) which allows for offloading eBPF to? Maybe I miss the
point again, but this is the only argument for bpfilter over nftables -
and that only if one ignores the option to implement an eBPF backend for
nftables VM). OK, maybe this clarifies once I know what you had in mind
when you wrote that reply.
Cheers, Phil
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:22 ` David Miller
2018-02-19 18:05 ` Phil Sutter
@ 2018-02-19 21:13 ` Florian Westphal
2018-02-20 10:44 ` Pablo Neira Ayuso
2 siblings, 0 replies; 73+ messages in thread
From: Florian Westphal @ 2018-02-19 21:13 UTC (permalink / raw)
To: David Miller
Cc: phil, laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
David Miller <davem@davemloft.net> wrote:
> From: Phil Sutter <phil@nwl.cc>
> Date: Mon, 19 Feb 2018 18:14:11 +0100
>
> > OK, so reading between the lines you're saying that nftables project
> > has failed to provide an adequate successor to iptables?
>
> Whilst it is great that the atomic table update problem was solved, I
> think the emphasis on flexibility often at the expense of performance
> was a bad move.
Thats not true, IMO.
One idea previosuly discussed was to add a 'freeze' option
to our nftables syntax. Essentially what would happen is that further
updates to the table become impossible, with exception of named sets
(which can be changed independently similar to ebpf maps is suppose).
As further updates to the table are then no longer allowed this would
then make it possible to e.g. jit all rules into a single program.
The table could still be removed (and recreated) of course so its
not impossible to make changes, but no longer at the rule level.
> Netfilter's chronic performance differential is why a lot of mindshare
> was lost to userspace networking technologies.
I think this is a unfair statement and also not true.
If you refer to the linear-ruleset-evaluation of iptables, this is
what ipset was added for.
Yes, its a band aid. But again, that problem come from the UAPI
format/limitations of only having one source or destination address per
rule, a limitation not present in nftables.
Other reason why iptables is a bit more costly than needed (although it
IS rather fast given, no spinlocks in main eval loop) are the rule
counter updates which were built into the design all those years ago.
Again, a problem solved in nftables by making the counters optional.
If you want to speedup forward path with XDP -- fine.
But AFAIU its still possible with XDP to have packets being sent to
full stack, right?
If so, it would be possible to even combine nftables with XDP, f.e.
by allowing an ebpf program running on host CPU to query netfilter
conntrack.
No Entry -> push to normal path
Entry -> check 'fastpath' flag (which would be in nf_conn struct).
Not set -> also normal path.
Otherwise continue XDP, stack bypass.
nftables would have a rule similar to this:
nft add rule inet forward ct state established ct label set fastpath
to switch such conntrack to xdp mode.
This decision can then be combined with nftables infra,
for example 'fatpath for tcp flows that saw more than 1mbit of data
in either direction' or the like.
Yes, this needs ebpf support for conntrack and NAT transformations,
and it does beg question how to handle the details, e.g. conntrack
timeouts. Don't see any unsolveable issues with this though.
Also has similarities with the 'flow offload' proposal, i.e. we
could perhaps even reuse what we already have to add provide flow
offload in software using epbf/XDP as offload backend.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 15:15 ` Florian Westphal
2018-02-19 15:27 ` David Miller
@ 2018-02-19 21:30 ` Jozsef Kadlecsik
1 sibling, 0 replies; 73+ messages in thread
From: Jozsef Kadlecsik @ 2018-02-19 21:30 UTC (permalink / raw)
To: Florian Westphal
Cc: David Miller, daniel, Harald Welte, netdev, netfilter-devel,
alexei.starovoitov
Hi David,
On Mon, 19 Feb 2018, Florian Westphal wrote:
> David Miller <davem@davemloft.net> wrote:
> >
> > Florian, first of all, the whole "change the iptables binary" idea is
> > a non-starter. For the many reasons I have described in the various
> > postings I have made today.
> >
> > It is entirely impractical.
You stressed several times that container images, virtualization
installations don't change - and that's exaggregation. Those are updated
as well, and not only because security updates must be rolled out, but
because new versions of softwares are requested.
You mentioned that the hosting part can upgrade the kernel - it means that
enabling NFTABLES is also a non-issue when the new eBPF functionality is
switched on, if that was missing.
> You suggest:
>
> iptables -> setsockopt -> umh (xtables -> ebpf) -> kernel
>
> How is this different from
>
> iptables -> setsockopt -> umh (Xtables -> nftables -> kernel
>
> ?
> EBPF can be placed within nftables either userspace or kernel,
> there is nothing that prevents this.
So why the second scenario suggested by Florian is not possible or must be
avoided? It not only could keep the unmodified iptables in the container
(if that's a must from some reason), but it would make possible to replace
it later anytime with iptables-compat/nftables.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-17 12:11 ` Harald Welte
2018-02-18 0:35 ` Florian Westphal
2018-02-19 12:03 ` Daniel Borkmann
@ 2018-02-19 21:42 ` Willem de Bruijn
2 siblings, 0 replies; 73+ messages in thread
From: Willem de Bruijn @ 2018-02-19 21:42 UTC (permalink / raw)
To: Harald Welte
Cc: Daniel Borkmann, Network Development, netfilter-devel,
David Miller, Alexei Starovoitov
> I see several possible areas of contention:
>
> 1) If you aim for a non-feature-complete support of iptables rules, it
> will create confusion to the users.
Right, you need full feature parity to be avoid ending up having to
maintain two implementations.
It seems uncontroversial that BPF can be very powerful if run at iptables
hooks. For performance, but also versatility. The android folks are converting
one out-of-tree module to BPF. There is probably a lot more such
business logic out there that is not suitable for inclusion in mainline as
an xt match/target, and that needs more access than xt_bpf can provide.
If a new first-class citizen BPF infra can do this and back the legacy
interface, too, that would save on maintenance. There is a steady
stream of fixes to iptables, e.g., from syzkaller vulnerability reports.
Just keeping the old implementation around as a dead letter is not a
safe deprecation strategy.
To bootstrap bpfilter, in the short term a reasonable set of iptables targets
and matches can perhaps be ported to BPF external functions with some
simple glue code.
> To me, this looks like some kind of legacy backwards compatibility
> mechanism that one would find in proprietary operating systems, but not
> in Linux. iptables, libiptc etc. are all free software. The source
> code can be edited, and you could just as well have a new version of
> iptables and/or libiptc which would pass the ruleset in userspace to
> your compiler, which would then insert the resulting eBPF program.
>
> Why add quite comprehensive kerne infrastructure? What's the motivation
> here?
The ABI deprecation point has been discussed quite a bit. If it is
infeasible to just drop the old interface, then an upcall mechanism
does seem the most practical approach to dynamically generating this
code. FWIW, as BPF is being used in more places, other locations
besides iptables could make use of this.
> Could you please clarify why the 'filter' table INPUT chain was used if
> you're using XDP? AFAICT they have completely different semantics.
>
> There is a well-conceived and generally understood notion of where
> exactly the filter/INPUT table processing happens. And that's not as
> early as in the NIC, but it's much later in the processing of the
> packet.
>
> I believe _if_ one wants to use the approach of "hiding" eBPF behind
> iptables, then either
>
> a) the eBPF programs must be executed at the exact same points in the
> stack as the existing hooks of the built-in chains of the
> filter/nat/mangle/raw tables, or
>
> b) you must introduce new 'tables', like an 'xdp' table which then has
> the notion of processing very early in processing, way before the
> normal filter table INPUT processing happens.
Agreed. One of the larger issues in the conversion of the Android
qtaguid conversion was the state surrounding the skb at the time of
processing. This example primarily depended on having skb->sk set.
Whether that is available at tc depends on early decap and even when
set the sk might prove different from the final one in the socket layer in
edge cases. Just one example how moving the call site can be very
fragile wrt state.
Another issue wrt moving around is availability of external functions
at different layers. XDP has access to far fewer than TC. For iptables,
I would imagine that you either want parity with TC or even a new
independent type. Parity would be useful also to expose some xt_match
functionality at the TC layer that is currently missing there.
> My main points are:
>
> 1) What is the goal of this?
My high bit feedback: for cases like taguid, it is very useful to be able
to execute BPF as drop-in at existing iptables locations, as is having
various match and target functionality available from BPF.
Maintaining the legacy ABI is basically dictated. If this can be achieved
while optimizing the runtime path and reducing maintenance that is
very appealing.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:09 ` Phil Sutter
2018-02-19 17:15 ` David Miller
@ 2018-02-20 9:35 ` Michal Kubecek
2018-02-20 18:10 ` Phil Sutter
1 sibling, 1 reply; 73+ messages in thread
From: Michal Kubecek @ 2018-02-20 9:35 UTC (permalink / raw)
To: Phil Sutter, David Miller, laforge, fw, daniel, netdev,
netfilter-devel, alexei.starovoitov
On Mon, Feb 19, 2018 at 06:09:39PM +0100, Phil Sutter wrote:
> What puzzles me about your argumentation is that you seem to propose for
> the kernel to cover up flaws in userspace. Spinning this concept further
> would mean that if there would be an old bug in iproute2 we should think
> of adding a workaround to rtnetlink interface in kernel because
> containers will keep the old iproute2 binary? Or am I (hopefully) just
> missing your point?
Actually, that's what we already do. This is from rtnl_dump_ifinfo():
/* A hack to preserve kernel<->userspace interface.
* The correct header is ifinfomsg. It is consistent with rtnl_getlink.
* However, before Linux v3.9 the code here assumed rtgenmsg and that's
* what iproute2 < v3.9.0 used.
* We can detect the old iproute2. Even including the IFLA_EXT_MASK
* attribute, its netlink message is shorter than struct ifinfomsg.
*/
(There are, in fact, even current tools using rtgenmsg but that's
another story.)
Michal Kubecek
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:22 ` David Miller
2018-02-19 18:05 ` Phil Sutter
2018-02-19 21:13 ` Florian Westphal
@ 2018-02-20 10:44 ` Pablo Neira Ayuso
2018-02-20 14:07 ` Daniel Borkmann
` (2 more replies)
2 siblings, 3 replies; 73+ messages in thread
From: Pablo Neira Ayuso @ 2018-02-20 10:44 UTC (permalink / raw)
To: David Miller
Cc: phil, laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David!
On Mon, Feb 19, 2018 at 12:22:26PM -0500, David Miller wrote:
[...]
> Netfilter's chronic performance differential is why a lot of mindshare
> was lost to userspace networking technologies.
Claiming that Netfilter is the reason for the massive adoption of
userspace networking isn't a fair statement at all.
Let's talk about performance if this is what you want:
* Our benchmarks here are delivering ~x9.5 performance boost for IPv4
load balancing from netfilter ingress.
* ~x2 faster than iptables prerouting when dropping packets at very
early stage in the network datapath - dos attack scenario - again from
the ingress hook.
* The new flowtable infrastructure that will show up in 4.16 provides
a faster forwarding path, measuring ~x2 faster forwarding here, _by
simply adding one single rule to your FORWARD chain_. And that's
just the initial implementation that got merged upstream, we have
room to fly even faster.
And that's just the beginning, we have more ongoing work, incrementally
based on top of what we have, to provide even faster datapath paths with
very simple configurations.
Note that those numbers above are very similar numbers to what we have
seen in bpf. Well, to be honest, we're just slightly behind bpf, since
benchmarks I have seen on loading balancing IPv4 is x10 from XDP,
dropping packets also slightly more than x2, which is actually happening
way earlier than ingress, naturally dropping earlier gives us better
numbers.
But it's not all about performance... let's have a look at the "iron
triangle"...
We keep usability in our radar, that's paramount for us. Netfilter is
probably so much widely more adopted than tc because of this. The kind
of problems that big Silicon datacenters have to face are simply
different to the millions of devices running Linux outthere, there are
plenty of smart devops outthere that sacrify the little performance loss
at the cost of keeping it easy to configure and maintain things.
If we want to talk about problems...
Every project has its own subset of problems. In that sense, anyone that
has spent time playing with the bpf infrastructure is very much aware of
all of its usability problems:
* You have to disable optimizations in llvm, otherwise the verifier
gets confused too smart compiler optimizations and rejects the code.
* Very hard to debug the reason why the verifier is rejecting apparently
valid code. That results in people playing strange "moving code around
up and down".
* Lack of sufficient abstraction: bpf is not only exposing its own
software bugs through its interface, but it will also bite the dust
with CPU bugs due to lack of glue code to hide details behind the
syscall interface curtain. That will need a kernel upgrade after all to
fix, so all benefits of adding new programs. We've even seem claims on
performance being more important than security in this mailing list.
Don't get me wrong, no software is safe from security issues, but if you
don't abstract your resources in the right way, you have more chance to
have experimence more problems.
Just to mention a few of them.
So, please, let's focus each of us in our own work. Let me remind your
wise words - I think just one year ago in another of these episodes of
the bpf vs. netfilter: "We're all working to achieve the same goals",
even if we're working on competing projects inside Linux.
Thanks!
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-19 17:15 ` David Miller
@ 2018-02-20 13:05 ` Phil Sutter
0 siblings, 0 replies; 73+ messages in thread
From: Phil Sutter @ 2018-02-20 13:05 UTC (permalink / raw)
To: David Miller
Cc: laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
Hi David,
On Mon, Feb 19, 2018 at 12:15:37PM -0500, David Miller wrote:
> From: Phil Sutter <phil@nwl.cc>
> Date: Mon, 19 Feb 2018 18:09:39 +0100
>
> > What puzzles me about your argumentation is that you seem to propose for
> > the kernel to cover up flaws in userspace. Spinning this concept further
> > would mean that if there would be an old bug in iproute2 we should think
> > of adding a workaround to rtnetlink interface in kernel because
> > containers will keep the old iproute2 binary? Or am I (hopefully) just
> > missing your point?
>
> I'll answer this with a question. I tried to remove UFO entirely from
> the kernel, did you see how that went?
:)
I didn't follow back then, but found mails about KVM live migration
breakage when moving to a kernel without UFO. But isn't that a problem
with how virtio_net optimizes things? Florian recently told me how
iptables CHECKSUM target was mainly introduced to overcome a different
problem in the same area. So all this is kernel covering up for kernel
problems. My question was about covering up for userspace bugs in
kernelspace. If you think that is preferable over fixing userspace, I
have to put that in consideration when dealing with userspace issues.
Cheers, Phil
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-20 10:44 ` Pablo Neira Ayuso
@ 2018-02-20 14:07 ` Daniel Borkmann
2018-02-20 14:55 ` David Miller
2018-02-21 1:52 ` Alexei Starovoitov
2 siblings, 0 replies; 73+ messages in thread
From: Daniel Borkmann @ 2018-02-20 14:07 UTC (permalink / raw)
To: Pablo Neira Ayuso, David Miller
Cc: phil, laforge, fw, netdev, netfilter-devel, alexei.starovoitov
On 02/20/2018 11:44 AM, Pablo Neira Ayuso wrote:
> Hi David!
>
> On Mon, Feb 19, 2018 at 12:22:26PM -0500, David Miller wrote:
> [...]
>> Netfilter's chronic performance differential is why a lot of mindshare
>> was lost to userspace networking technologies.
>
> Claiming that Netfilter is the reason for the massive adoption of
> userspace networking isn't a fair statement at all.
>
> Let's talk about performance if this is what you want:
>
> * Our benchmarks here are delivering ~x9.5 performance boost for IPv4
> load balancing from netfilter ingress.
>
> * ~x2 faster than iptables prerouting when dropping packets at very
> early stage in the network datapath - dos attack scenario - again from
> the ingress hook.
>
> * The new flowtable infrastructure that will show up in 4.16 provides
> a faster forwarding path, measuring ~x2 faster forwarding here, _by
> simply adding one single rule to your FORWARD chain_. And that's
> just the initial implementation that got merged upstream, we have
> room to fly even faster.
>
> And that's just the beginning, we have more ongoing work, incrementally
> based on top of what we have, to provide even faster datapath paths with
> very simple configurations.
>
> Note that those numbers above are very similar numbers to what we have
> seen in bpf. Well, to be honest, we're just slightly behind bpf, since
> benchmarks I have seen on loading balancing IPv4 is x10 from XDP,
> dropping packets also slightly more than x2, which is actually happening
> way earlier than ingress, naturally dropping earlier gives us better
> numbers.
>
> But it's not all about performance... let's have a look at the "iron
> triangle"...
>
> We keep usability in our radar, that's paramount for us. Netfilter is
> probably so much widely more adopted than tc because of this. The kind
Right, in terms of performance the above is what tc ingress used to do
already long ago after spinlock removal could be lifted, which was an
important step on that direction. In terms of usability, sure, it's always
a 'fun' topic on that matter for a number of classifier / actions mostly
from the older days. I think there it has improved a bit over time,
but at least speaking of things like cls_bpf, it's trivial to attach an
object somewhere via tc cmdline.
> of problems that big Silicon datacenters have to face are simply
> different to the millions of devices running Linux outthere, there are
> plenty of smart devops outthere that sacrify the little performance loss
> at the cost of keeping it easy to configure and maintain things.
>
> If we want to talk about problems...
>
> Every project has its own subset of problems. In that sense, anyone that
> has spent time playing with the bpf infrastructure is very much aware of
> all of its usability problems:
>
> * You have to disable optimizations in llvm, otherwise the verifier
> gets confused too smart compiler optimizations and rejects the code.
That is actually a false claim, which makes me think that you didn't even
give this a try at all before stating the above. Funny enough, for a very
long period of time in LLVM's BPF back end when you used other optimization
levels than the -O2, clang would bark with an internal error, for example:
$ clang-3.9 -target bpf -O0 -c foo.c -o /tmp/foo.o
fatal error: error in backend: Cannot select: 0x5633ae698280: ch,glue = BPFISD::CALL 0x5633ae698210, 0x5633ae697e90, Register:i64 %R1, Register:i64 %R2, Register:i64 %R3,
0x5633ae698210:1
0x5633ae697e90: i64,ch = load<LD8[@tail_call]> 0x5633ae6955e0, 0x5633ae694fc0, undef:i64
0x5633ae694fc0: i64 = BPFISD::Wrapper TargetGlobalAddress:i64<void (%struct.__sk_buff*, i8*, i32)** @tail_call> 0
[...]
Whereas -O2 *is* the general recommendation for everyone to use:
$ clang-3.9 -target bpf -O2 -c foo.c -o /tmp/foo.o
$
This is fixed in later versions, e.g. in clang-7.0 such back end error is
gone anyway fwiw. But in any case, we're running complex programs with -O2
optimization levels for several years now just fine. Yes, given we do push
BPF to the limits we had some corner cases where the verifier had to be
adjusted, but overall the number of cases reduced over time, which is also
a natural progression when people use it in various advanced ways. In fact,
it's a much better choice to use clang with -O2 here since simply the majority
of people use it that way. And if you consume it via higher level front ends
e.g. bcc, ply, bpftrace to name a few from tracing side, then you don't need
to care at all about this. (But in addition to that, there's also continuous
effort on LLVM side to optimize BPF code generation in various ways.)
> * Very hard to debug the reason why the verifier is rejecting apparently
> valid code. That results in people playing strange "moving code around
> up and down".
Please show me your programs and I'm happy to help you out. :-) Yes, in the
earlier days, I would consider it might have been hard; during the course
of the last few years, the verifier and LLVM back end have seen both heavy
improvements all over the place e.g. llvm-objdump correlating verifier errors
back to the pseudo C code via dwarf was a bigger one on the latter for example.
Writing BPF programs definitely became easier although there's always undoubted
room for improvement and the work we're heading towards will make it more
natural to develop programs in the C front end it provides, reducing further
potential contention with the verifier. It takes a bit to get used to the
verifier analysis, but then there's always a learning curve for getting into
new frameworks and develop a basic understanding for their semantics. Same
holds true when people would switch from using their known ip*tables-translate
syntax to using nft directly.
Anyway, aside from this, for BPF we also have the case that the people who
develop programs to solve problems with the help of this technology are just
a small subset of the ones that are using it. Best example is probably, as
mentioned some time ago in the thread, the hard work from Brendan Gregg and
many others in bcc to develop all the really easy to consume tracing cmdline
tools.
> * Lack of sufficient abstraction: bpf is not only exposing its own
> software bugs through its interface, but it will also bite the dust
> with CPU bugs due to lack of glue code to hide details behind the
> syscall interface curtain. That will need a kernel upgrade after all to
> fix, so all benefits of adding new programs. We've even seem claims on
> performance being more important than security in this mailing list.
> Don't get me wrong, no software is safe from security issues, but if you
> don't abstract your resources in the right way, you have more chance to
> have experimence more problems.
Sorry, but this is just nebulous FUD here. Yes, every software has bugs. If
there are bugs, we handle them and fix it, period. So? You've probably seen
the extensive kernel selftest suite we have developed over time that by now
contains more than over 1k test cases on the BPF core infrastructure and many
more to come. Quite frankly, I'm actually very happy on the progress from
the syzkaller folks in recent months as well to stress BPF continuously, and
it finds bugs just as well in other areas (like netfilter), so yeah, we all
do keep our heads down and fix them properly in order to make everything more
robust.
> Just to mention a few of them.
>
> So, please, let's focus each of us in our own work. Let me remind your
> wise words - I think just one year ago in another of these episodes of
> the bpf vs. netfilter: "We're all working to achieve the same goals",
> even if we're working on competing projects inside Linux.
>
> Thanks!
>
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-20 10:44 ` Pablo Neira Ayuso
2018-02-20 14:07 ` Daniel Borkmann
@ 2018-02-20 14:55 ` David Miller
2018-02-21 1:52 ` Alexei Starovoitov
2 siblings, 0 replies; 73+ messages in thread
From: David Miller @ 2018-02-20 14:55 UTC (permalink / raw)
To: pablo
Cc: phil, laforge, fw, daniel, netdev, netfilter-devel, alexei.starovoitov
From: Pablo Neira Ayuso <pablo@netfilter.org>
Date: Tue, 20 Feb 2018 11:44:31 +0100
> * Lack of sufficient abstraction: bpf is not only exposing its own
> software bugs through its interface, but it will also bite the dust
> with CPU bugs due to lack of glue code to hide details behind the
> syscall interface curtain. That will need a kernel upgrade after all to
> fix, so all benefits of adding new programs. We've even seem claims on
> performance being more important than security in this mailing list.
> Don't get me wrong, no software is safe from security issues, but if you
> don't abstract your resources in the right way, you have more chance to
> have experimence more problems.
I find it surprising that the person who didn't even know that
generating classical BPF was not appropriate in his patches is
suddenly a complete expert on eBPF and all of it's shortcomings.
Pablo, I am sincerely very disappointed in you, and if you continue
to attack eBPF in such an ignorant way going forward we will have
a very hard time taking you seriously at all.
Thank you.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-20 9:35 ` Michal Kubecek
@ 2018-02-20 18:10 ` Phil Sutter
0 siblings, 0 replies; 73+ messages in thread
From: Phil Sutter @ 2018-02-20 18:10 UTC (permalink / raw)
To: Michal Kubecek
Cc: David Miller, laforge, fw, daniel, netdev, netfilter-devel,
alexei.starovoitov
Hi Michal,
On Tue, Feb 20, 2018 at 10:35:41AM +0100, Michal Kubecek wrote:
> On Mon, Feb 19, 2018 at 06:09:39PM +0100, Phil Sutter wrote:
> > What puzzles me about your argumentation is that you seem to propose for
> > the kernel to cover up flaws in userspace. Spinning this concept further
> > would mean that if there would be an old bug in iproute2 we should think
> > of adding a workaround to rtnetlink interface in kernel because
> > containers will keep the old iproute2 binary? Or am I (hopefully) just
> > missing your point?
>
> Actually, that's what we already do. This is from rtnl_dump_ifinfo():
>
> /* A hack to preserve kernel<->userspace interface.
> * The correct header is ifinfomsg. It is consistent with rtnl_getlink.
> * However, before Linux v3.9 the code here assumed rtgenmsg and that's
> * what iproute2 < v3.9.0 used.
> * We can detect the old iproute2. Even including the IFLA_EXT_MASK
> * attribute, its netlink message is shorter than struct ifinfomsg.
> */
The reason why this is in place (and should be IMHO) is that commit
88c5b5ce5cb57 ("rtnetlink: Call nlmsg_parse() with correct header
length") incompatibly changed uAPI.
I have a different example which reflects what I have in mind, namely
iproute2 commit 33f6dd23a51c4 ("ip fou: pass family attribute as u8")
which basically does:
| - addattr16(n, 1024, FOU_ATTR_AF, family);
| + addattr8(n, 1024, FOU_ATTR_AF, family);
If kernel cares about those userspace bugs, shouldn't the better fix be
to make it expect u16 in FOU_ATTR_AF and check whether the high or low
byte contains the expected value?
Cheers, Phil
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-20 10:44 ` Pablo Neira Ayuso
2018-02-20 14:07 ` Daniel Borkmann
2018-02-20 14:55 ` David Miller
@ 2018-02-21 1:52 ` Alexei Starovoitov
2018-02-21 12:01 ` Pablo Neira Ayuso
2 siblings, 1 reply; 73+ messages in thread
From: Alexei Starovoitov @ 2018-02-21 1:52 UTC (permalink / raw)
To: Pablo Neira Ayuso
Cc: David Miller, phil, laforge, fw, daniel, netdev, netfilter-devel
On Tue, Feb 20, 2018 at 11:44:31AM +0100, Pablo Neira Ayuso wrote:
>
> Don't get me wrong, no software is safe from security issues, but if you
> don't abstract your resources in the right way, you have more chance to
> have experimence more problems.
interesting point.
The key part of iptables and nft design is heavy use of indirect calls.
The execution of single iptable rule is ~3 indirect calls.
Quite a lot worse in nft where every expression is an indirect call.
If my math is correct even simplest nft rule will get to ~10.
It was all fine until spectre2 was discovered and retpoline
now adds 20-30 cycles for each indirect call.
To put numbers in perspective the simple
for(...)
indirect_call();
loop without retpoline does ~500 M iterations per second on 2+Ghz xeon.
clang -mretpoline
gcc -mindirect-branch=thunk
gcc -mindirect-branch=thunk-inline
produce slightly different code with performance of 80-90 M
iterations per second for the above loop.
Looks like iptables/nft did not abstract the resources in
the right way and now experiences more problems.
CPUs will eventually be fixed and IBRS_ALL will become reality.
Until then the kernel has to deal with the performance issues.
bpf and the networking stack will suffer from retpoline as well
and we need to work asap on devirtualization and other ideas.
For xdp a single indirect call we do per packet (to call into bpf prog)
is noticeable and we're experimenting with static_key-like approach to
call bpf program with direct call.
bpf_tail_calls will suffer too and cannot be accelerated as-is.
To solve that we're working on dynamic linking via verifier improvements.
C based bpf programs will use normal indirect calls, but verifier will
replace indirect with direct at pointer update time.
It's not going to be easy, but bpf and stack is fixable,
whereas iptables/nft are going to suffer until fixed CPUs find
their way into servers years from now.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-21 1:52 ` Alexei Starovoitov
@ 2018-02-21 12:01 ` Pablo Neira Ayuso
2018-02-21 12:13 ` Florian Westphal
0 siblings, 1 reply; 73+ messages in thread
From: Pablo Neira Ayuso @ 2018-02-21 12:01 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: David Miller, phil, laforge, fw, daniel, netdev, netfilter-devel
On Tue, Feb 20, 2018 at 05:52:54PM -0800, Alexei Starovoitov wrote:
> On Tue, Feb 20, 2018 at 11:44:31AM +0100, Pablo Neira Ayuso wrote:
> >
> > Don't get me wrong, no software is safe from security issues, but if you
> > don't abstract your resources in the right way, you have more chance to
> > have experimence more problems.
>
> interesting point.
> The key part of iptables and nft design is heavy use of indirect calls.
> The execution of single iptable rule is ~3 indirect calls.
> Quite a lot worse in nft where every expression is an indirect call.
That's right. Netfilter is probably too modular, probably we can
revisit this to find a better balance, actually Felix Fietkau was
recently rising concerns on this, specifically in environments with
limited space to store the kernel image. We'll have a look, thanks for
remind us about this.
[...]
> CPUs will eventually be fixed and IBRS_ALL will become reality.
> Until then the kernel has to deal with the performance issues.
Hopefully, so we can all skip these problems.
Thanks.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: [PATCH RFC 0/4] net: add bpfilter
2018-02-21 12:01 ` Pablo Neira Ayuso
@ 2018-02-21 12:13 ` Florian Westphal
2018-02-22 2:20 ` nft/bpf interpreters and spectre2. Was: " Alexei Starovoitov
0 siblings, 1 reply; 73+ messages in thread
From: Florian Westphal @ 2018-02-21 12:13 UTC (permalink / raw)
To: Pablo Neira Ayuso
Cc: Alexei Starovoitov, David Miller, phil, laforge, fw, daniel,
netdev, netfilter-devel
Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On Tue, Feb 20, 2018 at 05:52:54PM -0800, Alexei Starovoitov wrote:
> > On Tue, Feb 20, 2018 at 11:44:31AM +0100, Pablo Neira Ayuso wrote:
> > >
> > > Don't get me wrong, no software is safe from security issues, but if you
> > > don't abstract your resources in the right way, you have more chance to
> > > have experimence more problems.
> >
> > interesting point.
> > The key part of iptables and nft design is heavy use of indirect calls.
> > The execution of single iptable rule is ~3 indirect calls.
> > Quite a lot worse in nft where every expression is an indirect call.
>
> That's right. Netfilter is probably too modular, probably we can
> revisit this to find a better balance, actually Felix Fietkau was
> recently rising concerns on this, specifically in environments with
> limited space to store the kernel image. We'll have a look, thanks for
> remind us about this.
Agree, we have too many config knobs, probably a good idea to turn some
modules into plain .o (like cmp and bitwise).
Obvious candidates are: meta, numgen, limit, objref, quota, reject.
We should probably also consider removing
CONFIG_NFT_SET_RBTREE and CONFIG_NFT_SET_HASH and just always
build both too (at least rbtree since that offers interval).
For the indirect call issue we can use direct calls from eval loop for
some of the more frequently used ones, similar to what we do already
for nft_cmp_fast_expr. But maybe we don't even have to if we can
get help to build a jitter that takes an nftables netlink table dump
and builds jit code from that.
^ permalink raw reply [flat|nested] 73+ messages in thread
* nft/bpf interpreters and spectre2. Was: [PATCH RFC 0/4] net: add bpfilter
2018-02-21 12:13 ` Florian Westphal
@ 2018-02-22 2:20 ` Alexei Starovoitov
2018-02-22 11:39 ` Pablo Neira Ayuso
0 siblings, 1 reply; 73+ messages in thread
From: Alexei Starovoitov @ 2018-02-22 2:20 UTC (permalink / raw)
To: Florian Westphal, jannh
Cc: Pablo Neira Ayuso, David Miller, phil, laforge, daniel, netdev,
netfilter-devel
On Wed, Feb 21, 2018 at 01:13:03PM +0100, Florian Westphal wrote:
>
> Obvious candidates are: meta, numgen, limit, objref, quota, reject.
>
> We should probably also consider removing
> CONFIG_NFT_SET_RBTREE and CONFIG_NFT_SET_HASH and just always
> build both too (at least rbtree since that offers interval).
>
> For the indirect call issue we can use direct calls from eval loop for
> some of the more frequently used ones, similar to what we do already
> for nft_cmp_fast_expr.
nft_cmp_fast_expr and other expressions mentioned above made me thinking...
do we have the same issue with nft interpreter as we had with bpf one?
bpf interpreter was used as part of spectre2 attack to leak
information via cache side channel and let VM read hypervisor memory.
Due to that issue we removed bpf interpreter from the kernel code.
That's what CONFIG_BPF_JIT_ALWAYS_ON for...
but we still have nft interpreter in the kernel that can also
execute arbitrary nft expressions.
Jann's exploit used the following bpf instructions:
struct bpf_insn evil_bytecode_instrs[] = {
// rax = target_byte_addr
{ .code = BPF_LD | BPF_IMM | BPF_DW, .dst_reg = 0, .imm = target_byte_addr }, { .imm = target_byte_addr>>32 },
// rdi = timing_leak_array
{ .code = BPF_LD | BPF_IMM | BPF_DW, .dst_reg = 1, .imm = host_timing_leak_addr }, { .imm = host_timing_leak_addr>>32 },
// rax = *(u8*)rax
{ .code = BPF_LDX | BPF_MEM | BPF_B, .dst_reg = 0, .src_reg = 0, .off = 0 },
// rax = rax << ...
{ .code = BPF_ALU64 | BPF_LSH | BPF_K, .dst_reg = 0, .imm = 10 - bit_idx },
// rax = rax & 0x400
{ .code = BPF_ALU64 | BPF_AND | BPF_K, .dst_reg = 0, .imm = 0x400 },
// rax = rdi + rax
{ .code = BPF_ALU64 | BPF_ADD | BPF_X, .dst_reg = 0, .src_reg = 1 },
// *(u8*) (rax + 0x800)
{ .code = BPF_LDX | BPF_MEM | BPF_B, .dst_reg = 0, .src_reg = 0, .off = 0x800 },
and a gadget to jump into __bpf_prog_run with insn pointing
to memory controlled by the guest while accessible
(at different virt address) by the hypervisor.
It seems possible to construct similar sequence of instructions
out of nft expressions and use gadget that jumps into nft_do_chain().
The attacker would need to discover more kernel addresses:
nft_do_chain, nft_cmp_fast_ops, nft_payload_fast_ops, nft_bitwise_eval,
nft_lookup_eval, and nft_bitmap_lookup
to populate nft chains, rules and expressions in guest memory
comparing to bpf interpreter attack.
Then in nft_do_chain(struct nft_pktinfo *pkt, void *priv)
pkt needs to point to fake struct sk_buff in guest memory with
skb->head == target_byte_addr
The first nft expression can be nft_payload_fast_eval().
If it's properly constructed with
(nft_payload->based == NFT_PAYLOAD_NETWORK_HEADER, offset == 0, len == 0, dreg == 1)
it will do arbitrary load of
*(u8 *)dest = *(u8 *)ptr;
from target_byte_addr into register 1 of nft state machine
(dest is u32 array of registers in the stack of nft_do_chain)
Second nft expression can be nft_bitwise_eval() to mask particular
bit in register 1.
Then nft_cmp_eval() to check whether bit is one or zero and
conditional NFT_BREAK out of first nft expression into second nft rule.
The last conditional nft_immediate_eval() in the first rule will set
register 1 to 0x400 * 8 while the first nft_bitwise_eval() in
the second rule with do r1 &= 0x400 * 8.
So at this point r1 will have either 0x400 * 8 or 0 depending
on value of speculatively loaded bit.
The last expression can be nft_lookup_eval() with
nft_lookup->set->ops->lookup == nft_bitmap_lookup
which will do nft_bitmap->bitmap[idx] where idx = r1 / 8
The memory used for this last nft_lookup/bitmap expression is
both an instruction and timing_leak_array itself.
If I'm not mistaken, this sequence of nft expression will
speculatively execute very similar logic as in evil_bytecode_instrs[]
The amount of actual speculative native cpu load/stores/branches is
probably more than executed by bpf interpreter for these evil bytecodes,
but likely well within cpu speculation window of 100+ insns.
Obviously such exploit is harder to do than bpf based one.
Do we need to do anything about it ?
May be it's easier to find gadgets in .text of vmlinux
instead of messing with interpreters?
Jann,
can you comment on removing interpreters in general?
Do we need to worry about having bpf and/or nft interpreter
in the kernel?
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: nft/bpf interpreters and spectre2. Was: [PATCH RFC 0/4] net: add bpfilter
2018-02-22 2:20 ` nft/bpf interpreters and spectre2. Was: " Alexei Starovoitov
@ 2018-02-22 11:39 ` Pablo Neira Ayuso
2018-02-22 17:06 ` Alexei Starovoitov
0 siblings, 1 reply; 73+ messages in thread
From: Pablo Neira Ayuso @ 2018-02-22 11:39 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Florian Westphal, jannh, David Miller, phil, laforge, daniel,
netdev, netfilter-devel
Hi Alexei,
On Wed, Feb 21, 2018 at 06:20:37PM -0800, Alexei Starovoitov wrote:
> On Wed, Feb 21, 2018 at 01:13:03PM +0100, Florian Westphal wrote:
> >
> > Obvious candidates are: meta, numgen, limit, objref, quota, reject.
> >
> > We should probably also consider removing
> > CONFIG_NFT_SET_RBTREE and CONFIG_NFT_SET_HASH and just always
> > build both too (at least rbtree since that offers interval).
> >
> > For the indirect call issue we can use direct calls from eval loop for
> > some of the more frequently used ones, similar to what we do already
> > for nft_cmp_fast_expr.
>
> nft_cmp_fast_expr and other expressions mentioned above made me thinking...
>
> do we have the same issue with nft interpreter as we had with bpf one?
> bpf interpreter was used as part of spectre2 attack to leak
> information via cache side channel and let VM read hypervisor memory.
> Due to that issue we removed bpf interpreter from the kernel code.
> That's what CONFIG_BPF_JIT_ALWAYS_ON for...
> but we still have nft interpreter in the kernel that can also
> execute arbitrary nft expressions.
>
> Jann's exploit used the following bpf instructions:
> struct bpf_insn evil_bytecode_instrs[] = {
> // rax = target_byte_addr
> { .code = BPF_LD | BPF_IMM | BPF_DW, .dst_reg = 0, .imm = target_byte_addr }, { .imm = target_byte_addr>>32 },
We don't place pointers in the nft VM registers, it's basically
illegal to do so, otherwise we would need more sophisticated verifier.
I'm telling this because we don't have a way to point to any arbitrary
address as in 'target_byte_addr' above.
> // rdi = timing_leak_array
> { .code = BPF_LD | BPF_IMM | BPF_DW, .dst_reg = 1, .imm = host_timing_leak_addr }, { .imm = host_timing_leak_addr>>32 },
> // rax = *(u8*)rax
> { .code = BPF_LDX | BPF_MEM | BPF_B, .dst_reg = 0, .src_reg = 0, .off = 0 },
> // rax = rax << ...
> { .code = BPF_ALU64 | BPF_LSH | BPF_K, .dst_reg = 0, .imm = 10 - bit_idx },
> // rax = rax & 0x400
> { .code = BPF_ALU64 | BPF_AND | BPF_K, .dst_reg = 0, .imm = 0x400 },
> // rax = rdi + rax
> { .code = BPF_ALU64 | BPF_ADD | BPF_X, .dst_reg = 0, .src_reg = 1 },
> // *(u8*) (rax + 0x800)
> { .code = BPF_LDX | BPF_MEM | BPF_B, .dst_reg = 0, .src_reg = 0, .off = 0x800 },
>
> and a gadget to jump into __bpf_prog_run with insn pointing
> to memory controlled by the guest while accessible
> (at different virt address) by the hypervisor.
>
> It seems possible to construct similar sequence of instructions
> out of nft expressions and use gadget that jumps into nft_do_chain().
> The attacker would need to discover more kernel addresses:
> nft_do_chain, nft_cmp_fast_ops, nft_payload_fast_ops, nft_bitwise_eval,
> nft_lookup_eval, and nft_bitmap_lookup
> to populate nft chains, rules and expressions in guest memory
> comparing to bpf interpreter attack.
>
> Then in nft_do_chain(struct nft_pktinfo *pkt, void *priv)
> pkt needs to point to fake struct sk_buff in guest memory with
> skb->head == target_byte_addr
We don't have a way to make this point to fake struct sk_buff.
> The first nft expression can be nft_payload_fast_eval().
> If it's properly constructed with
> (nft_payload->based == NFT_PAYLOAD_NETWORK_HEADER, offset == 0, len == 0, dreg == 1)
We can reject len == 0. To be honest, this is not done right now, but
we can place a patch to validate this. Given this is a specialized
networking virtual machine, it retain semantics, so fetching zero
length data from a skbuff makes no sense, hence, we can return EINVAL
via netlink when adding a rule that tries to do this.
> it will do arbitrary load of
> *(u8 *)dest = *(u8 *)ptr;
> from target_byte_addr into register 1 of nft state machine
> (dest is u32 array of registers in the stack of nft_do_chain)
> Second nft expression can be nft_bitwise_eval() to mask particular
> bit in register 1.
> Then nft_cmp_eval() to check whether bit is one or zero and
> conditional NFT_BREAK out of first nft expression into second nft rule.
> The last conditional nft_immediate_eval() in the first rule will set
> register 1 to 0x400 * 8 while the first nft_bitwise_eval() in
> the second rule with do r1 &= 0x400 * 8.
> So at this point r1 will have either 0x400 * 8 or 0 depending
> on value of speculatively loaded bit.
> The last expression can be nft_lookup_eval() with
> nft_lookup->set->ops->lookup == nft_bitmap_lookup
> which will do nft_bitmap->bitmap[idx] where idx = r1 / 8
> The memory used for this last nft_lookup/bitmap expression is
> both an instruction and timing_leak_array itself.
> If I'm not mistaken, this sequence of nft expression will
> speculatively execute very similar logic as in evil_bytecode_instrs[]
My impression is that several assumptions above are not correct.
> The amount of actual speculative native cpu load/stores/branches is
> probably more than executed by bpf interpreter for these evil bytecodes,
> but likely well within cpu speculation window of 100+ insns.
>
> Obviously such exploit is harder to do than bpf based one.
> Do we need to do anything about it ?
> May be it's easier to find gadgets in .text of vmlinux
> instead of messing with interpreters?
>
> Jann,
> can you comment on removing interpreters in general?
> Do we need to worry about having bpf and/or nft interpreter
> in the kernel?
Jann, any review on that front is very much welcome, so please let me
know if you find any real problem.
Thanks for reviewing!
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: nft/bpf interpreters and spectre2. Was: [PATCH RFC 0/4] net: add bpfilter
2018-02-22 11:39 ` Pablo Neira Ayuso
@ 2018-02-22 17:06 ` Alexei Starovoitov
2018-02-22 18:47 ` Jann Horn
0 siblings, 1 reply; 73+ messages in thread
From: Alexei Starovoitov @ 2018-02-22 17:06 UTC (permalink / raw)
To: Pablo Neira Ayuso
Cc: Florian Westphal, jannh, David Miller, phil, laforge, daniel,
netdev, netfilter-devel
On Thu, Feb 22, 2018 at 12:39:15PM +0100, Pablo Neira Ayuso wrote:
> Hi Alexei,
>
> On Wed, Feb 21, 2018 at 06:20:37PM -0800, Alexei Starovoitov wrote:
> > On Wed, Feb 21, 2018 at 01:13:03PM +0100, Florian Westphal wrote:
> > >
> > > Obvious candidates are: meta, numgen, limit, objref, quota, reject.
> > >
> > > We should probably also consider removing
> > > CONFIG_NFT_SET_RBTREE and CONFIG_NFT_SET_HASH and just always
> > > build both too (at least rbtree since that offers interval).
> > >
> > > For the indirect call issue we can use direct calls from eval loop for
> > > some of the more frequently used ones, similar to what we do already
> > > for nft_cmp_fast_expr.
> >
> > nft_cmp_fast_expr and other expressions mentioned above made me thinking...
> >
> > do we have the same issue with nft interpreter as we had with bpf one?
> > bpf interpreter was used as part of spectre2 attack to leak
> > information via cache side channel and let VM read hypervisor memory.
> > Due to that issue we removed bpf interpreter from the kernel code.
> > That's what CONFIG_BPF_JIT_ALWAYS_ON for...
> > but we still have nft interpreter in the kernel that can also
> > execute arbitrary nft expressions.
> >
> > Jann's exploit used the following bpf instructions:
> > struct bpf_insn evil_bytecode_instrs[] = {
> > // rax = target_byte_addr
> > { .code = BPF_LD | BPF_IMM | BPF_DW, .dst_reg = 0, .imm = target_byte_addr }, { .imm = target_byte_addr>>32 },
>
> We don't place pointers in the nft VM registers, it's basically
> illegal to do so, otherwise we would need more sophisticated verifier.
> I'm telling this because we don't have a way to point to any arbitrary
> address as in 'target_byte_addr' above.
these evil_bytecode_instrs never saw bpf verifier either.
That's the scary part of that poc.
The only requirement for poc to work is to have interpreter
in executable part of hypervisor code and speculatively jump into it
with arguments pointing to memory controlled by vm.
All static checks (done by bpf verifier and by nft validation) are bypassed.
The only way to defend from such exploit is either remove the interpreter
from the kernel or add _run-time_ checks and masks for every memory access
(similar to what is done for spectre1 mitigations).
In case of bpf it's impractical.
In case of nft I suspect so too. I don't yet see how nft can check
that skb pointer passed as part of nft_pktinfo is not an actual skb.
> > // rdi = timing_leak_array
> > { .code = BPF_LD | BPF_IMM | BPF_DW, .dst_reg = 1, .imm = host_timing_leak_addr }, { .imm = host_timing_leak_addr>>32 },
> > // rax = *(u8*)rax
> > { .code = BPF_LDX | BPF_MEM | BPF_B, .dst_reg = 0, .src_reg = 0, .off = 0 },
> > // rax = rax << ...
> > { .code = BPF_ALU64 | BPF_LSH | BPF_K, .dst_reg = 0, .imm = 10 - bit_idx },
> > // rax = rax & 0x400
> > { .code = BPF_ALU64 | BPF_AND | BPF_K, .dst_reg = 0, .imm = 0x400 },
> > // rax = rdi + rax
> > { .code = BPF_ALU64 | BPF_ADD | BPF_X, .dst_reg = 0, .src_reg = 1 },
> > // *(u8*) (rax + 0x800)
> > { .code = BPF_LDX | BPF_MEM | BPF_B, .dst_reg = 0, .src_reg = 0, .off = 0x800 },
> >
> > and a gadget to jump into __bpf_prog_run with insn pointing
> > to memory controlled by the guest while accessible
> > (at different virt address) by the hypervisor.
> >
> > It seems possible to construct similar sequence of instructions
> > out of nft expressions and use gadget that jumps into nft_do_chain().
> > The attacker would need to discover more kernel addresses:
> > nft_do_chain, nft_cmp_fast_ops, nft_payload_fast_ops, nft_bitwise_eval,
> > nft_lookup_eval, and nft_bitmap_lookup
> > to populate nft chains, rules and expressions in guest memory
> > comparing to bpf interpreter attack.
> >
> > Then in nft_do_chain(struct nft_pktinfo *pkt, void *priv)
> > pkt needs to point to fake struct sk_buff in guest memory with
> > skb->head == target_byte_addr
>
> We don't have a way to make this point to fake struct sk_buff.
yet. it's possible, since cpu is speculating and all such pointers
controlled by vm can be arbitrary.
> > The first nft expression can be nft_payload_fast_eval().
> > If it's properly constructed with
> > (nft_payload->based == NFT_PAYLOAD_NETWORK_HEADER, offset == 0, len == 0, dreg == 1)
>
> We can reject len == 0. To be honest, this is not done right now, but
> we can place a patch to validate this. Given this is a specialized
> networking virtual machine, it retain semantics, so fetching zero
> length data from a skbuff makes no sense, hence, we can return EINVAL
> via netlink when adding a rule that tries to do this.
Adding static check won't help.
^ permalink raw reply [flat|nested] 73+ messages in thread
* Re: nft/bpf interpreters and spectre2. Was: [PATCH RFC 0/4] net: add bpfilter
2018-02-22 17:06 ` Alexei Starovoitov
@ 2018-02-22 18:47 ` Jann Horn
0 siblings, 0 replies; 73+ messages in thread
From: Jann Horn @ 2018-02-22 18:47 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Pablo Neira Ayuso, Florian Westphal, David Miller, phil, laforge,
Daniel Borkmann, Network Development, netfilter-devel
[resend as plaintext, apparently mobile gmail will send HTML mails]
On Thu, Feb 22, 2018 at 3:20 AM, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
> On Wed, Feb 21, 2018 at 01:13:03PM +0100, Florian Westphal wrote:
>>
>> Obvious candidates are: meta, numgen, limit, objref, quota, reject.
>>
>> We should probably also consider removing
>> CONFIG_NFT_SET_RBTREE and CONFIG_NFT_SET_HASH and just always
>> build both too (at least rbtree since that offers interval).
>>
>> For the indirect call issue we can use direct calls from eval loop for
>> some of the more frequently used ones, similar to what we do already
>> for nft_cmp_fast_expr.
>
> nft_cmp_fast_expr and other expressions mentioned above made me thinking...
>
> do we have the same issue with nft interpreter as we had with bpf one?
> bpf interpreter was used as part of spectre2 attack to leak
> information via cache side channel and let VM read hypervisor memory.
> Due to that issue we removed bpf interpreter from the kernel code.
> That's what CONFIG_BPF_JIT_ALWAYS_ON for...
> but we still have nft interpreter in the kernel that can also
> execute arbitrary nft expressions.
>
> Jann's exploit used the following bpf instructions:
[...]
>
> and a gadget to jump into __bpf_prog_run with insn pointing
> to memory controlled by the guest while accessible
> (at different virt address) by the hypervisor.
>
> It seems possible to construct similar sequence of instructions
> out of nft expressions and use gadget that jumps into nft_do_chain().
[...]
> Obviously such exploit is harder to do than bpf based one.
> Do we need to do anything about it ?
> May be it's easier to find gadgets in .text of vmlinux
> instead of messing with interpreters?
>
> Jann,
> can you comment on removing interpreters in general?
> Do we need to worry about having bpf and/or nft interpreter
> in the kernel?
I think that for Spectre V2, the presence of interpreters isn't a big
problem. It simplifies writing attacks a bit, but I don't expect it to
be necessary if an attacker invests some time into finding useful
gadgets.
^ permalink raw reply [flat|nested] 73+ messages in thread
end of thread, other threads:[~2018-02-22 18:54 UTC | newest]
Thread overview: 73+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-16 13:40 [PATCH RFC 0/4] net: add bpfilter Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 1/4] modules: allow insmod load regular elf binaries Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 2/4] bpf: introduce bpfilter commands Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 3/4] net: initial bpfilter skeleton Daniel Borkmann
2018-02-16 13:40 ` [PATCH RFC 4/4] bpf: rough bpfilter codegen example hack Daniel Borkmann
2018-02-16 14:57 ` [PATCH RFC 0/4] net: add bpfilter Florian Westphal
2018-02-16 16:14 ` Florian Westphal
2018-02-16 20:44 ` Daniel Borkmann
2018-02-17 12:33 ` Harald Welte
2018-02-17 19:18 ` Florian Westphal
2018-02-16 22:33 ` David Miller
2018-02-17 12:21 ` Harald Welte
2018-02-17 20:10 ` Florian Westphal
2018-02-17 22:38 ` Florian Westphal
2018-02-16 16:53 ` Daniel Borkmann
2018-02-16 22:32 ` David Miller
2018-02-17 12:11 ` Harald Welte
2018-02-18 0:35 ` Florian Westphal
2018-02-19 12:03 ` Daniel Borkmann
2018-02-19 12:52 ` Harald Welte
2018-02-19 14:44 ` David Miller
2018-02-19 14:53 ` Florian Westphal
2018-02-19 15:07 ` David Miller
2018-02-19 15:20 ` Florian Westphal
2018-02-19 15:28 ` David Miller
2018-02-19 15:23 ` Harald Welte
2018-02-19 15:32 ` David Miller
2018-02-19 15:37 ` Jan Engelhardt
2018-02-19 15:43 ` David Miller
2018-02-19 15:36 ` David Miller
2018-02-19 17:20 ` Harald Welte
2018-02-19 17:29 ` David Miller
2018-02-19 18:37 ` Harald Welte
2018-02-19 18:47 ` David Miller
2018-02-19 17:40 ` Arturo Borrero Gonzalez
2018-02-19 18:06 ` Arturo Borrero Gonzalez
2018-02-19 18:43 ` David Miller
2018-02-19 15:00 ` David Miller
2018-02-19 14:59 ` Florian Westphal
2018-02-19 15:13 ` David Miller
2018-02-19 15:15 ` Florian Westphal
2018-02-19 15:27 ` David Miller
2018-02-19 15:38 ` Harald Welte
2018-02-19 15:44 ` David Miller
2018-02-19 17:14 ` Phil Sutter
2018-02-19 17:22 ` David Miller
2018-02-19 18:05 ` Phil Sutter
2018-02-19 18:41 ` David Miller
2018-02-19 20:41 ` Phil Sutter
2018-02-19 21:13 ` Florian Westphal
2018-02-20 10:44 ` Pablo Neira Ayuso
2018-02-20 14:07 ` Daniel Borkmann
2018-02-20 14:55 ` David Miller
2018-02-21 1:52 ` Alexei Starovoitov
2018-02-21 12:01 ` Pablo Neira Ayuso
2018-02-21 12:13 ` Florian Westphal
2018-02-22 2:20 ` nft/bpf interpreters and spectre2. Was: " Alexei Starovoitov
2018-02-22 11:39 ` Pablo Neira Ayuso
2018-02-22 17:06 ` Alexei Starovoitov
2018-02-22 18:47 ` Jann Horn
2018-02-19 17:41 ` Arturo Borrero Gonzalez
2018-02-19 21:30 ` Jozsef Kadlecsik
2018-02-19 15:27 ` Harald Welte
2018-02-19 15:31 ` David Miller
2018-02-19 17:09 ` Phil Sutter
2018-02-19 17:15 ` David Miller
2018-02-20 13:05 ` Phil Sutter
2018-02-20 9:35 ` Michal Kubecek
2018-02-20 18:10 ` Phil Sutter
2018-02-19 17:32 ` Harald Welte
2018-02-19 17:41 ` Arturo Borrero Gonzalez
2018-02-19 21:42 ` Willem de Bruijn
2018-02-18 23:35 ` Florian Westphal
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.