* [PATCH ipvs-next] ipvs: count pre-established TCP states as active
@ 2016-06-03 15:56 Michal Kubecek
2016-06-06 7:23 ` Julian Anastasov
2016-06-12 15:27 ` Julian Anastasov
0 siblings, 2 replies; 4+ messages in thread
From: Michal Kubecek @ 2016-06-03 15:56 UTC (permalink / raw)
To: lvs-devel
Cc: Wensong Zhang, Simon Horman, Julian Anastasov, netfilter-devel,
coreteam, netdev, linux-kernel, Pablo Neira Ayuso,
Patrick McHardy, Jozsef Kadlecsik, David S. Miller
Some users observed that "least connection" distribution algorithm doesn't
handle well bursts of TCP connections from reconnecting clients after
a node or network failure.
This is because the algorithm counts active connection as worth 256
inactive ones where for TCP, "active" only means TCP connections in
ESTABLISHED state. In case of a connection burst, new connections are
handled before previous ones have finished the three way handshaking so
that all are still counted as "inactive", i.e. cheap ones. The become
"active" quickly but at that time, all of them are already assigned to one
real server (or few), resulting in highly unbalanced distribution.
Address this by counting the "pre-established" states as "active".
Signed-off-by: Michal Kubecek <mkubecek@suse.cz>
---
net/netfilter/ipvs/ip_vs_proto_tcp.c | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
diff --git a/net/netfilter/ipvs/ip_vs_proto_tcp.c b/net/netfilter/ipvs/ip_vs_proto_tcp.c
index d7024b2ed769..5117bcb7d2f0 100644
--- a/net/netfilter/ipvs/ip_vs_proto_tcp.c
+++ b/net/netfilter/ipvs/ip_vs_proto_tcp.c
@@ -395,6 +395,20 @@ static const char *const tcp_state_name_table[IP_VS_TCP_S_LAST+1] = {
[IP_VS_TCP_S_LAST] = "BUG!",
};
+static const bool tcp_state_active_table[IP_VS_TCP_S_LAST] = {
+ [IP_VS_TCP_S_NONE] = false,
+ [IP_VS_TCP_S_ESTABLISHED] = true,
+ [IP_VS_TCP_S_SYN_SENT] = true,
+ [IP_VS_TCP_S_SYN_RECV] = true,
+ [IP_VS_TCP_S_FIN_WAIT] = false,
+ [IP_VS_TCP_S_TIME_WAIT] = false,
+ [IP_VS_TCP_S_CLOSE] = false,
+ [IP_VS_TCP_S_CLOSE_WAIT] = false,
+ [IP_VS_TCP_S_LAST_ACK] = false,
+ [IP_VS_TCP_S_LISTEN] = false,
+ [IP_VS_TCP_S_SYNACK] = true,
+};
+
#define sNO IP_VS_TCP_S_NONE
#define sES IP_VS_TCP_S_ESTABLISHED
#define sSS IP_VS_TCP_S_SYN_SENT
@@ -418,6 +432,13 @@ static const char * tcp_state_name(int state)
return tcp_state_name_table[state] ? tcp_state_name_table[state] : "?";
}
+static bool tcp_state_active(int state)
+{
+ if (state >= IP_VS_TCP_S_LAST)
+ return false;
+ return tcp_state_active_table[state];
+}
+
static struct tcp_states_t tcp_states [] = {
/* INPUT */
/* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
@@ -540,12 +561,12 @@ set_tcp_state(struct ip_vs_proto_data *pd, struct ip_vs_conn *cp,
if (dest) {
if (!(cp->flags & IP_VS_CONN_F_INACTIVE) &&
- (new_state != IP_VS_TCP_S_ESTABLISHED)) {
+ !tcp_state_active(new_state)) {
atomic_dec(&dest->activeconns);
atomic_inc(&dest->inactconns);
cp->flags |= IP_VS_CONN_F_INACTIVE;
} else if ((cp->flags & IP_VS_CONN_F_INACTIVE) &&
- (new_state == IP_VS_TCP_S_ESTABLISHED)) {
+ tcp_state_active(new_state)) {
atomic_inc(&dest->activeconns);
atomic_dec(&dest->inactconns);
cp->flags &= ~IP_VS_CONN_F_INACTIVE;
--
2.8.3
^ permalink raw reply related [flat|nested] 4+ messages in thread* Re: [PATCH ipvs-next] ipvs: count pre-established TCP states as active
2016-06-03 15:56 [PATCH ipvs-next] ipvs: count pre-established TCP states as active Michal Kubecek
@ 2016-06-06 7:23 ` Julian Anastasov
2016-06-12 15:27 ` Julian Anastasov
1 sibling, 0 replies; 4+ messages in thread
From: Julian Anastasov @ 2016-06-06 7:23 UTC (permalink / raw)
To: Michal Kubecek
Cc: lvs-devel, Wensong Zhang, Simon Horman, netfilter-devel,
coreteam, netdev, linux-kernel, Pablo Neira Ayuso,
Patrick McHardy, Jozsef Kadlecsik, David S. Miller
Hello,
On Fri, 3 Jun 2016, Michal Kubecek wrote:
> Some users observed that "least connection" distribution algorithm doesn't
> handle well bursts of TCP connections from reconnecting clients after
> a node or network failure.
>
> This is because the algorithm counts active connection as worth 256
> inactive ones where for TCP, "active" only means TCP connections in
> ESTABLISHED state. In case of a connection burst, new connections are
> handled before previous ones have finished the three way handshaking so
> that all are still counted as "inactive", i.e. cheap ones. The become
> "active" quickly but at that time, all of them are already assigned to one
> real server (or few), resulting in highly unbalanced distribution.
>
> Address this by counting the "pre-established" states as "active".
>
> Signed-off-by: Michal Kubecek <mkubecek@suse.cz>
LC and WLC are bursty by nature. May be a new
scheduler is needed that combines the LC algorithm with
WRR mode to adaptively reduce the difference in load,
especially for the case when new server is started in
a setup with many servers.
Give me some days or week to analyze the effects of
your patch, mostly in situations of SYN attack. Note that
there can be schedulers affected by this change but I think
they will only benefit from it.
For example:
- LBLC, LBLCR: can use weight as threshold for activeconns
- NQ, SED: only activeconns are used
- OVF: weight is used as threshold for activeconns,
inactconns are not used
Schedulers not part of the kernel can be affacted
too. So, for now the plan is to apply this patch. If there
are other opinions, please speak up.
> ---
> net/netfilter/ipvs/ip_vs_proto_tcp.c | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/net/netfilter/ipvs/ip_vs_proto_tcp.c b/net/netfilter/ipvs/ip_vs_proto_tcp.c
> index d7024b2ed769..5117bcb7d2f0 100644
> --- a/net/netfilter/ipvs/ip_vs_proto_tcp.c
> +++ b/net/netfilter/ipvs/ip_vs_proto_tcp.c
> @@ -395,6 +395,20 @@ static const char *const tcp_state_name_table[IP_VS_TCP_S_LAST+1] = {
> [IP_VS_TCP_S_LAST] = "BUG!",
> };
>
> +static const bool tcp_state_active_table[IP_VS_TCP_S_LAST] = {
> + [IP_VS_TCP_S_NONE] = false,
> + [IP_VS_TCP_S_ESTABLISHED] = true,
> + [IP_VS_TCP_S_SYN_SENT] = true,
> + [IP_VS_TCP_S_SYN_RECV] = true,
> + [IP_VS_TCP_S_FIN_WAIT] = false,
> + [IP_VS_TCP_S_TIME_WAIT] = false,
> + [IP_VS_TCP_S_CLOSE] = false,
> + [IP_VS_TCP_S_CLOSE_WAIT] = false,
> + [IP_VS_TCP_S_LAST_ACK] = false,
> + [IP_VS_TCP_S_LISTEN] = false,
> + [IP_VS_TCP_S_SYNACK] = true,
> +};
> +
> #define sNO IP_VS_TCP_S_NONE
> #define sES IP_VS_TCP_S_ESTABLISHED
> #define sSS IP_VS_TCP_S_SYN_SENT
> @@ -418,6 +432,13 @@ static const char * tcp_state_name(int state)
> return tcp_state_name_table[state] ? tcp_state_name_table[state] : "?";
> }
>
> +static bool tcp_state_active(int state)
> +{
> + if (state >= IP_VS_TCP_S_LAST)
> + return false;
> + return tcp_state_active_table[state];
> +}
> +
> static struct tcp_states_t tcp_states [] = {
> /* INPUT */
> /* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
> @@ -540,12 +561,12 @@ set_tcp_state(struct ip_vs_proto_data *pd, struct ip_vs_conn *cp,
>
> if (dest) {
> if (!(cp->flags & IP_VS_CONN_F_INACTIVE) &&
> - (new_state != IP_VS_TCP_S_ESTABLISHED)) {
> + !tcp_state_active(new_state)) {
> atomic_dec(&dest->activeconns);
> atomic_inc(&dest->inactconns);
> cp->flags |= IP_VS_CONN_F_INACTIVE;
> } else if ((cp->flags & IP_VS_CONN_F_INACTIVE) &&
> - (new_state == IP_VS_TCP_S_ESTABLISHED)) {
> + tcp_state_active(new_state)) {
> atomic_inc(&dest->activeconns);
> atomic_dec(&dest->inactconns);
> cp->flags &= ~IP_VS_CONN_F_INACTIVE;
> --
> 2.8.3
Regards
--
Julian Anastasov <ja@ssi.bg>
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH ipvs-next] ipvs: count pre-established TCP states as active
2016-06-03 15:56 [PATCH ipvs-next] ipvs: count pre-established TCP states as active Michal Kubecek
2016-06-06 7:23 ` Julian Anastasov
@ 2016-06-12 15:27 ` Julian Anastasov
2016-06-13 5:20 ` Simon Horman
1 sibling, 1 reply; 4+ messages in thread
From: Julian Anastasov @ 2016-06-12 15:27 UTC (permalink / raw)
To: Michal Kubecek
Cc: lvs-devel, Wensong Zhang, Simon Horman, netfilter-devel,
coreteam, netdev, linux-kernel, Pablo Neira Ayuso,
Patrick McHardy, Jozsef Kadlecsik, David S. Miller
Hello,
On Fri, 3 Jun 2016, Michal Kubecek wrote:
> Some users observed that "least connection" distribution algorithm doesn't
> handle well bursts of TCP connections from reconnecting clients after
> a node or network failure.
>
> This is because the algorithm counts active connection as worth 256
> inactive ones where for TCP, "active" only means TCP connections in
> ESTABLISHED state. In case of a connection burst, new connections are
> handled before previous ones have finished the three way handshaking so
> that all are still counted as "inactive", i.e. cheap ones. The become
> "active" quickly but at that time, all of them are already assigned to one
> real server (or few), resulting in highly unbalanced distribution.
>
> Address this by counting the "pre-established" states as "active".
>
> Signed-off-by: Michal Kubecek <mkubecek@suse.cz>
Acked-by: Julian Anastasov <ja@ssi.bg>
Simon, please apply!
> ---
> net/netfilter/ipvs/ip_vs_proto_tcp.c | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/net/netfilter/ipvs/ip_vs_proto_tcp.c b/net/netfilter/ipvs/ip_vs_proto_tcp.c
> index d7024b2ed769..5117bcb7d2f0 100644
> --- a/net/netfilter/ipvs/ip_vs_proto_tcp.c
> +++ b/net/netfilter/ipvs/ip_vs_proto_tcp.c
> @@ -395,6 +395,20 @@ static const char *const tcp_state_name_table[IP_VS_TCP_S_LAST+1] = {
> [IP_VS_TCP_S_LAST] = "BUG!",
> };
>
> +static const bool tcp_state_active_table[IP_VS_TCP_S_LAST] = {
> + [IP_VS_TCP_S_NONE] = false,
> + [IP_VS_TCP_S_ESTABLISHED] = true,
> + [IP_VS_TCP_S_SYN_SENT] = true,
> + [IP_VS_TCP_S_SYN_RECV] = true,
> + [IP_VS_TCP_S_FIN_WAIT] = false,
> + [IP_VS_TCP_S_TIME_WAIT] = false,
> + [IP_VS_TCP_S_CLOSE] = false,
> + [IP_VS_TCP_S_CLOSE_WAIT] = false,
> + [IP_VS_TCP_S_LAST_ACK] = false,
> + [IP_VS_TCP_S_LISTEN] = false,
> + [IP_VS_TCP_S_SYNACK] = true,
> +};
> +
> #define sNO IP_VS_TCP_S_NONE
> #define sES IP_VS_TCP_S_ESTABLISHED
> #define sSS IP_VS_TCP_S_SYN_SENT
> @@ -418,6 +432,13 @@ static const char * tcp_state_name(int state)
> return tcp_state_name_table[state] ? tcp_state_name_table[state] : "?";
> }
>
> +static bool tcp_state_active(int state)
> +{
> + if (state >= IP_VS_TCP_S_LAST)
> + return false;
> + return tcp_state_active_table[state];
> +}
> +
> static struct tcp_states_t tcp_states [] = {
> /* INPUT */
> /* sNO, sES, sSS, sSR, sFW, sTW, sCL, sCW, sLA, sLI, sSA */
> @@ -540,12 +561,12 @@ set_tcp_state(struct ip_vs_proto_data *pd, struct ip_vs_conn *cp,
>
> if (dest) {
> if (!(cp->flags & IP_VS_CONN_F_INACTIVE) &&
> - (new_state != IP_VS_TCP_S_ESTABLISHED)) {
> + !tcp_state_active(new_state)) {
> atomic_dec(&dest->activeconns);
> atomic_inc(&dest->inactconns);
> cp->flags |= IP_VS_CONN_F_INACTIVE;
> } else if ((cp->flags & IP_VS_CONN_F_INACTIVE) &&
> - (new_state == IP_VS_TCP_S_ESTABLISHED)) {
> + tcp_state_active(new_state)) {
> atomic_inc(&dest->activeconns);
> atomic_dec(&dest->inactconns);
> cp->flags &= ~IP_VS_CONN_F_INACTIVE;
> --
> 2.8.3
Regards
--
Julian Anastasov <ja@ssi.bg>
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH ipvs-next] ipvs: count pre-established TCP states as active
2016-06-12 15:27 ` Julian Anastasov
@ 2016-06-13 5:20 ` Simon Horman
0 siblings, 0 replies; 4+ messages in thread
From: Simon Horman @ 2016-06-13 5:20 UTC (permalink / raw)
To: Julian Anastasov
Cc: Michal Kubecek, lvs-devel, Wensong Zhang, netfilter-devel,
coreteam, netdev, linux-kernel, Pablo Neira Ayuso,
Patrick McHardy, Jozsef Kadlecsik, David S. Miller
On Sun, Jun 12, 2016 at 06:27:39PM +0300, Julian Anastasov wrote:
>
> Hello,
>
> On Fri, 3 Jun 2016, Michal Kubecek wrote:
>
> > Some users observed that "least connection" distribution algorithm doesn't
> > handle well bursts of TCP connections from reconnecting clients after
> > a node or network failure.
> >
> > This is because the algorithm counts active connection as worth 256
> > inactive ones where for TCP, "active" only means TCP connections in
> > ESTABLISHED state. In case of a connection burst, new connections are
> > handled before previous ones have finished the three way handshaking so
> > that all are still counted as "inactive", i.e. cheap ones. The become
> > "active" quickly but at that time, all of them are already assigned to one
> > real server (or few), resulting in highly unbalanced distribution.
> >
> > Address this by counting the "pre-established" states as "active".
> >
> > Signed-off-by: Michal Kubecek <mkubecek@suse.cz>
>
> Acked-by: Julian Anastasov <ja@ssi.bg>
>
> Simon, please apply!
Thanks, done.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-06-13 5:20 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-03 15:56 [PATCH ipvs-next] ipvs: count pre-established TCP states as active Michal Kubecek
2016-06-06 7:23 ` Julian Anastasov
2016-06-12 15:27 ` Julian Anastasov
2016-06-13 5:20 ` Simon Horman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).