* [PATCH net-next v4] net: ipvs: add sysctl_run_estimation to support disable estimation
@ 2021-08-20 5:37 Dust Li
2021-08-21 8:41 ` Julian Anastasov
0 siblings, 1 reply; 4+ messages in thread
From: Dust Li @ 2021-08-20 5:37 UTC (permalink / raw)
To: Julian Anastasov, Simon Horman, Wensong Zhang
Cc: lvs-devel, netdev, yunhong-cgl jiang
estimation_timer will iterate the est_list to do estimation
for each ipvs stats. When there are lots of services, the
list can be very large.
We found that estimation_timer() run for more then 200ms on a
machine with 104 CPU and 50K services.
yunhong-cgl jiang report the same phenomenon before:
https://www.spinics.net/lists/lvs-devel/msg05426.html
In some cases(for example a large K8S cluster with many ipvs services),
ipvs estimation may not be needed. So adding a sysctl blob to allow
users to disable this completely.
Default is: 1 (enable)
Cc: yunhong-cgl jiang <xintian1976@gmail.com>
Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
---
v2: Use common sysctl facilities
v3: Fix sysctl_run_estimation() redefine when CONFIG_SYSCTL not enabled
v4: Some typo and minor fixes
Documentation/networking/ipvs-sysctl.rst | 11 +++++++++++
include/net/ip_vs.h | 11 +++++++++++
net/netfilter/ipvs/ip_vs_ctl.c | 8 ++++++++
net/netfilter/ipvs/ip_vs_est.c | 5 +++++
4 files changed, 35 insertions(+)
diff --git a/Documentation/networking/ipvs-sysctl.rst b/Documentation/networking/ipvs-sysctl.rst
index 2afccc63856e..95ef56d62077 100644
--- a/Documentation/networking/ipvs-sysctl.rst
+++ b/Documentation/networking/ipvs-sysctl.rst
@@ -300,3 +300,14 @@ sync_version - INTEGER
Kernels with this sync_version entry are able to receive messages
of both version 1 and version 2 of the synchronisation protocol.
+
+run_estimation - BOOLEAN
+ 0 - disabled
+ not 0 - enabled (default)
+
+ If disabled, the estimation will be stop, and you can't see
+ any update on speed estimation data.
+
+ You can always re-enable estimation by setting this value to 1.
+ But be careful, the first estimation after re-enable is not
+ accurate.
diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h
index 7cb5a1aace40..ff1804a0c469 100644
--- a/include/net/ip_vs.h
+++ b/include/net/ip_vs.h
@@ -931,6 +931,7 @@ struct netns_ipvs {
int sysctl_conn_reuse_mode;
int sysctl_schedule_icmp;
int sysctl_ignore_tunneled;
+ int sysctl_run_estimation;
/* ip_vs_lblc */
int sysctl_lblc_expiration;
@@ -1071,6 +1072,11 @@ static inline int sysctl_cache_bypass(struct netns_ipvs *ipvs)
return ipvs->sysctl_cache_bypass;
}
+static inline int sysctl_run_estimation(struct netns_ipvs *ipvs)
+{
+ return ipvs->sysctl_run_estimation;
+}
+
#else
static inline int sysctl_sync_threshold(struct netns_ipvs *ipvs)
@@ -1163,6 +1169,11 @@ static inline int sysctl_cache_bypass(struct netns_ipvs *ipvs)
return 0;
}
+static inline int sysctl_run_estimation(struct netns_ipvs *ipvs)
+{
+ return 1;
+}
+
#endif
/* IPVS core functions
diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
index c25097092a06..cbea5a68afb5 100644
--- a/net/netfilter/ipvs/ip_vs_ctl.c
+++ b/net/netfilter/ipvs/ip_vs_ctl.c
@@ -2017,6 +2017,12 @@ static struct ctl_table vs_vars[] = {
.mode = 0644,
.proc_handler = proc_dointvec,
},
+ {
+ .procname = "run_estimation",
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec,
+ },
#ifdef CONFIG_IP_VS_DEBUG
{
.procname = "debug_level",
@@ -4090,6 +4096,8 @@ static int __net_init ip_vs_control_net_init_sysctl(struct netns_ipvs *ipvs)
tbl[idx++].data = &ipvs->sysctl_conn_reuse_mode;
tbl[idx++].data = &ipvs->sysctl_schedule_icmp;
tbl[idx++].data = &ipvs->sysctl_ignore_tunneled;
+ ipvs->sysctl_run_estimation = 1;
+ tbl[idx++].data = &ipvs->sysctl_run_estimation;
ipvs->sysctl_hdr = register_net_sysctl(net, "net/ipv4/vs", tbl);
if (ipvs->sysctl_hdr == NULL) {
diff --git a/net/netfilter/ipvs/ip_vs_est.c b/net/netfilter/ipvs/ip_vs_est.c
index 05b8112ffb37..9a1a7af6a186 100644
--- a/net/netfilter/ipvs/ip_vs_est.c
+++ b/net/netfilter/ipvs/ip_vs_est.c
@@ -100,6 +100,9 @@ static void estimation_timer(struct timer_list *t)
u64 rate;
struct netns_ipvs *ipvs = from_timer(ipvs, t, est_timer);
+ if (!sysctl_run_estimation(ipvs))
+ goto skip;
+
spin_lock(&ipvs->est_lock);
list_for_each_entry(e, &ipvs->est_list, list) {
s = container_of(e, struct ip_vs_stats, est);
@@ -131,6 +134,8 @@ static void estimation_timer(struct timer_list *t)
spin_unlock(&s->lock);
}
spin_unlock(&ipvs->est_lock);
+
+skip:
mod_timer(&ipvs->est_timer, jiffies + 2*HZ);
}
--
2.19.1.3.ge56e4f7
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH net-next v4] net: ipvs: add sysctl_run_estimation to support disable estimation
2021-08-20 5:37 [PATCH net-next v4] net: ipvs: add sysctl_run_estimation to support disable estimation Dust Li
@ 2021-08-21 8:41 ` Julian Anastasov
2021-10-02 8:59 ` Simon Horman
0 siblings, 1 reply; 4+ messages in thread
From: Julian Anastasov @ 2021-08-21 8:41 UTC (permalink / raw)
To: Dust Li
Cc: Simon Horman, Wensong Zhang, lvs-devel, Pablo Neira Ayuso,
netfilter-devel, netdev, yunhong-cgl jiang
Hello,
On Fri, 20 Aug 2021, Dust Li wrote:
> estimation_timer will iterate the est_list to do estimation
> for each ipvs stats. When there are lots of services, the
> list can be very large.
> We found that estimation_timer() run for more then 200ms on a
> machine with 104 CPU and 50K services.
>
> yunhong-cgl jiang report the same phenomenon before:
> https://www.spinics.net/lists/lvs-devel/msg05426.html
>
> In some cases(for example a large K8S cluster with many ipvs services),
> ipvs estimation may not be needed. So adding a sysctl blob to allow
> users to disable this completely.
>
> Default is: 1 (enable)
>
> Cc: yunhong-cgl jiang <xintian1976@gmail.com>
> Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
Looks good to me, thanks!
Acked-by: Julian Anastasov <ja@ssi.bg>
> ---
> v2: Use common sysctl facilities
> v3: Fix sysctl_run_estimation() redefine when CONFIG_SYSCTL not enabled
> v4: Some typo and minor fixes
>
> Documentation/networking/ipvs-sysctl.rst | 11 +++++++++++
> include/net/ip_vs.h | 11 +++++++++++
> net/netfilter/ipvs/ip_vs_ctl.c | 8 ++++++++
> net/netfilter/ipvs/ip_vs_est.c | 5 +++++
> 4 files changed, 35 insertions(+)
>
> diff --git a/Documentation/networking/ipvs-sysctl.rst b/Documentation/networking/ipvs-sysctl.rst
> index 2afccc63856e..95ef56d62077 100644
> --- a/Documentation/networking/ipvs-sysctl.rst
> +++ b/Documentation/networking/ipvs-sysctl.rst
> @@ -300,3 +300,14 @@ sync_version - INTEGER
>
> Kernels with this sync_version entry are able to receive messages
> of both version 1 and version 2 of the synchronisation protocol.
> +
> +run_estimation - BOOLEAN
> + 0 - disabled
> + not 0 - enabled (default)
> +
> + If disabled, the estimation will be stop, and you can't see
> + any update on speed estimation data.
> +
> + You can always re-enable estimation by setting this value to 1.
> + But be careful, the first estimation after re-enable is not
> + accurate.
> diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h
> index 7cb5a1aace40..ff1804a0c469 100644
> --- a/include/net/ip_vs.h
> +++ b/include/net/ip_vs.h
> @@ -931,6 +931,7 @@ struct netns_ipvs {
> int sysctl_conn_reuse_mode;
> int sysctl_schedule_icmp;
> int sysctl_ignore_tunneled;
> + int sysctl_run_estimation;
>
> /* ip_vs_lblc */
> int sysctl_lblc_expiration;
> @@ -1071,6 +1072,11 @@ static inline int sysctl_cache_bypass(struct netns_ipvs *ipvs)
> return ipvs->sysctl_cache_bypass;
> }
>
> +static inline int sysctl_run_estimation(struct netns_ipvs *ipvs)
> +{
> + return ipvs->sysctl_run_estimation;
> +}
> +
> #else
>
> static inline int sysctl_sync_threshold(struct netns_ipvs *ipvs)
> @@ -1163,6 +1169,11 @@ static inline int sysctl_cache_bypass(struct netns_ipvs *ipvs)
> return 0;
> }
>
> +static inline int sysctl_run_estimation(struct netns_ipvs *ipvs)
> +{
> + return 1;
> +}
> +
> #endif
>
> /* IPVS core functions
> diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
> index c25097092a06..cbea5a68afb5 100644
> --- a/net/netfilter/ipvs/ip_vs_ctl.c
> +++ b/net/netfilter/ipvs/ip_vs_ctl.c
> @@ -2017,6 +2017,12 @@ static struct ctl_table vs_vars[] = {
> .mode = 0644,
> .proc_handler = proc_dointvec,
> },
> + {
> + .procname = "run_estimation",
> + .maxlen = sizeof(int),
> + .mode = 0644,
> + .proc_handler = proc_dointvec,
> + },
> #ifdef CONFIG_IP_VS_DEBUG
> {
> .procname = "debug_level",
> @@ -4090,6 +4096,8 @@ static int __net_init ip_vs_control_net_init_sysctl(struct netns_ipvs *ipvs)
> tbl[idx++].data = &ipvs->sysctl_conn_reuse_mode;
> tbl[idx++].data = &ipvs->sysctl_schedule_icmp;
> tbl[idx++].data = &ipvs->sysctl_ignore_tunneled;
> + ipvs->sysctl_run_estimation = 1;
> + tbl[idx++].data = &ipvs->sysctl_run_estimation;
>
> ipvs->sysctl_hdr = register_net_sysctl(net, "net/ipv4/vs", tbl);
> if (ipvs->sysctl_hdr == NULL) {
> diff --git a/net/netfilter/ipvs/ip_vs_est.c b/net/netfilter/ipvs/ip_vs_est.c
> index 05b8112ffb37..9a1a7af6a186 100644
> --- a/net/netfilter/ipvs/ip_vs_est.c
> +++ b/net/netfilter/ipvs/ip_vs_est.c
> @@ -100,6 +100,9 @@ static void estimation_timer(struct timer_list *t)
> u64 rate;
> struct netns_ipvs *ipvs = from_timer(ipvs, t, est_timer);
>
> + if (!sysctl_run_estimation(ipvs))
> + goto skip;
> +
> spin_lock(&ipvs->est_lock);
> list_for_each_entry(e, &ipvs->est_list, list) {
> s = container_of(e, struct ip_vs_stats, est);
> @@ -131,6 +134,8 @@ static void estimation_timer(struct timer_list *t)
> spin_unlock(&s->lock);
> }
> spin_unlock(&ipvs->est_lock);
> +
> +skip:
> mod_timer(&ipvs->est_timer, jiffies + 2*HZ);
> }
>
> --
> 2.19.1.3.ge56e4f7
>
>
>
Regards
--
Julian Anastasov <ja@ssi.bg>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH net-next v4] net: ipvs: add sysctl_run_estimation to support disable estimation
2021-08-21 8:41 ` Julian Anastasov
@ 2021-10-02 8:59 ` Simon Horman
2021-10-07 17:53 ` Pablo Neira Ayuso
0 siblings, 1 reply; 4+ messages in thread
From: Simon Horman @ 2021-10-02 8:59 UTC (permalink / raw)
To: Julian Anastasov, Pablo Neira Ayuso
Cc: Dust Li, Wensong Zhang, lvs-devel, netfilter-devel, netdev,
yunhong-cgl jiang
On Sat, Aug 21, 2021 at 11:41:50AM +0300, Julian Anastasov wrote:
>
> Hello,
>
> On Fri, 20 Aug 2021, Dust Li wrote:
>
> > estimation_timer will iterate the est_list to do estimation
> > for each ipvs stats. When there are lots of services, the
> > list can be very large.
> > We found that estimation_timer() run for more then 200ms on a
> > machine with 104 CPU and 50K services.
> >
> > yunhong-cgl jiang report the same phenomenon before:
> > https://www.spinics.net/lists/lvs-devel/msg05426.html
> >
> > In some cases(for example a large K8S cluster with many ipvs services),
> > ipvs estimation may not be needed. So adding a sysctl blob to allow
> > users to disable this completely.
> >
> > Default is: 1 (enable)
> >
> > Cc: yunhong-cgl jiang <xintian1976@gmail.com>
> > Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
>
> Looks good to me, thanks!
>
> Acked-by: Julian Anastasov <ja@ssi.bg>
Likwewise, thanks. And sorry for the delay.
Acked-by: Simon Horman <horms@verge.net.au>
Pablo, could you consider picking this up?
> > ---
> > v2: Use common sysctl facilities
> > v3: Fix sysctl_run_estimation() redefine when CONFIG_SYSCTL not enabled
> > v4: Some typo and minor fixes
> >
> > Documentation/networking/ipvs-sysctl.rst | 11 +++++++++++
> > include/net/ip_vs.h | 11 +++++++++++
> > net/netfilter/ipvs/ip_vs_ctl.c | 8 ++++++++
> > net/netfilter/ipvs/ip_vs_est.c | 5 +++++
> > 4 files changed, 35 insertions(+)
> >
> > diff --git a/Documentation/networking/ipvs-sysctl.rst b/Documentation/networking/ipvs-sysctl.rst
> > index 2afccc63856e..95ef56d62077 100644
> > --- a/Documentation/networking/ipvs-sysctl.rst
> > +++ b/Documentation/networking/ipvs-sysctl.rst
> > @@ -300,3 +300,14 @@ sync_version - INTEGER
> >
> > Kernels with this sync_version entry are able to receive messages
> > of both version 1 and version 2 of the synchronisation protocol.
> > +
> > +run_estimation - BOOLEAN
> > + 0 - disabled
> > + not 0 - enabled (default)
> > +
> > + If disabled, the estimation will be stop, and you can't see
> > + any update on speed estimation data.
> > +
> > + You can always re-enable estimation by setting this value to 1.
> > + But be careful, the first estimation after re-enable is not
> > + accurate.
> > diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h
> > index 7cb5a1aace40..ff1804a0c469 100644
> > --- a/include/net/ip_vs.h
> > +++ b/include/net/ip_vs.h
> > @@ -931,6 +931,7 @@ struct netns_ipvs {
> > int sysctl_conn_reuse_mode;
> > int sysctl_schedule_icmp;
> > int sysctl_ignore_tunneled;
> > + int sysctl_run_estimation;
> >
> > /* ip_vs_lblc */
> > int sysctl_lblc_expiration;
> > @@ -1071,6 +1072,11 @@ static inline int sysctl_cache_bypass(struct netns_ipvs *ipvs)
> > return ipvs->sysctl_cache_bypass;
> > }
> >
> > +static inline int sysctl_run_estimation(struct netns_ipvs *ipvs)
> > +{
> > + return ipvs->sysctl_run_estimation;
> > +}
> > +
> > #else
> >
> > static inline int sysctl_sync_threshold(struct netns_ipvs *ipvs)
> > @@ -1163,6 +1169,11 @@ static inline int sysctl_cache_bypass(struct netns_ipvs *ipvs)
> > return 0;
> > }
> >
> > +static inline int sysctl_run_estimation(struct netns_ipvs *ipvs)
> > +{
> > + return 1;
> > +}
> > +
> > #endif
> >
> > /* IPVS core functions
> > diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
> > index c25097092a06..cbea5a68afb5 100644
> > --- a/net/netfilter/ipvs/ip_vs_ctl.c
> > +++ b/net/netfilter/ipvs/ip_vs_ctl.c
> > @@ -2017,6 +2017,12 @@ static struct ctl_table vs_vars[] = {
> > .mode = 0644,
> > .proc_handler = proc_dointvec,
> > },
> > + {
> > + .procname = "run_estimation",
> > + .maxlen = sizeof(int),
> > + .mode = 0644,
> > + .proc_handler = proc_dointvec,
> > + },
> > #ifdef CONFIG_IP_VS_DEBUG
> > {
> > .procname = "debug_level",
> > @@ -4090,6 +4096,8 @@ static int __net_init ip_vs_control_net_init_sysctl(struct netns_ipvs *ipvs)
> > tbl[idx++].data = &ipvs->sysctl_conn_reuse_mode;
> > tbl[idx++].data = &ipvs->sysctl_schedule_icmp;
> > tbl[idx++].data = &ipvs->sysctl_ignore_tunneled;
> > + ipvs->sysctl_run_estimation = 1;
> > + tbl[idx++].data = &ipvs->sysctl_run_estimation;
> >
> > ipvs->sysctl_hdr = register_net_sysctl(net, "net/ipv4/vs", tbl);
> > if (ipvs->sysctl_hdr == NULL) {
> > diff --git a/net/netfilter/ipvs/ip_vs_est.c b/net/netfilter/ipvs/ip_vs_est.c
> > index 05b8112ffb37..9a1a7af6a186 100644
> > --- a/net/netfilter/ipvs/ip_vs_est.c
> > +++ b/net/netfilter/ipvs/ip_vs_est.c
> > @@ -100,6 +100,9 @@ static void estimation_timer(struct timer_list *t)
> > u64 rate;
> > struct netns_ipvs *ipvs = from_timer(ipvs, t, est_timer);
> >
> > + if (!sysctl_run_estimation(ipvs))
> > + goto skip;
> > +
> > spin_lock(&ipvs->est_lock);
> > list_for_each_entry(e, &ipvs->est_list, list) {
> > s = container_of(e, struct ip_vs_stats, est);
> > @@ -131,6 +134,8 @@ static void estimation_timer(struct timer_list *t)
> > spin_unlock(&s->lock);
> > }
> > spin_unlock(&ipvs->est_lock);
> > +
> > +skip:
> > mod_timer(&ipvs->est_timer, jiffies + 2*HZ);
> > }
> >
> > --
> > 2.19.1.3.ge56e4f7
> >
> >
> >
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH net-next v4] net: ipvs: add sysctl_run_estimation to support disable estimation
2021-10-02 8:59 ` Simon Horman
@ 2021-10-07 17:53 ` Pablo Neira Ayuso
0 siblings, 0 replies; 4+ messages in thread
From: Pablo Neira Ayuso @ 2021-10-07 17:53 UTC (permalink / raw)
To: Simon Horman
Cc: Julian Anastasov, Dust Li, Wensong Zhang, lvs-devel,
netfilter-devel, netdev, yunhong-cgl jiang
On Sat, Oct 02, 2021 at 10:59:29AM +0200, Simon Horman wrote:
> On Sat, Aug 21, 2021 at 11:41:50AM +0300, Julian Anastasov wrote:
> >
> > Hello,
> >
> > On Fri, 20 Aug 2021, Dust Li wrote:
> >
> > > estimation_timer will iterate the est_list to do estimation
> > > for each ipvs stats. When there are lots of services, the
> > > list can be very large.
> > > We found that estimation_timer() run for more then 200ms on a
> > > machine with 104 CPU and 50K services.
> > >
> > > yunhong-cgl jiang report the same phenomenon before:
> > > https://www.spinics.net/lists/lvs-devel/msg05426.html
> > >
> > > In some cases(for example a large K8S cluster with many ipvs services),
> > > ipvs estimation may not be needed. So adding a sysctl blob to allow
> > > users to disable this completely.
> > >
> > > Default is: 1 (enable)
> > >
> > > Cc: yunhong-cgl jiang <xintian1976@gmail.com>
> > > Signed-off-by: Dust Li <dust.li@linux.alibaba.com>
> >
> > Looks good to me, thanks!
> >
> > Acked-by: Julian Anastasov <ja@ssi.bg>
>
> Likwewise, thanks. And sorry for the delay.
>
> Acked-by: Simon Horman <horms@verge.net.au>
>
> Pablo, could you consider picking this up?
Applied, thanks.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-10-07 17:53 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-20 5:37 [PATCH net-next v4] net: ipvs: add sysctl_run_estimation to support disable estimation Dust Li
2021-08-21 8:41 ` Julian Anastasov
2021-10-02 8:59 ` Simon Horman
2021-10-07 17:53 ` Pablo Neira Ayuso
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.