From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: "Peter Zijlstra (Intel)" <peterz@infradead.org>,
Scott Wood <swood@redhat.com>, Ingo Molnar <mingo@kernel.org>,
Sasha Levin <sashal@kernel.org>
Subject: [PATCH AUTOSEL 5.5 04/58] timers/nohz: Update NOHZ load in remote tick
Date: Sat, 22 Feb 2020 21:20:25 -0500 [thread overview]
Message-ID: <20200223022119.707-4-sashal@kernel.org> (raw)
In-Reply-To: <20200223022119.707-1-sashal@kernel.org>
From: "Peter Zijlstra (Intel)" <peterz@infradead.org>
[ Upstream commit ebc0f83c78a2d26384401ecf2d2fa48063c0ee27 ]
The way loadavg is tracked during nohz only pays attention to the load
upon entering nohz. This can be particularly noticeable if full nohz is
entered while non-idle, and then the cpu goes idle and stays that way for
a long time.
Use the remote tick to ensure that full nohz cpus report their deltas
within a reasonable time.
[ swood: Added changelog and removed recheck of stopped tick. ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Scott Wood <swood@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lkml.kernel.org/r/1578736419-14628-3-git-send-email-swood@redhat.com
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
include/linux/sched/nohz.h | 2 ++
kernel/sched/core.c | 4 +++-
kernel/sched/loadavg.c | 33 +++++++++++++++++++++++----------
3 files changed, 28 insertions(+), 11 deletions(-)
diff --git a/include/linux/sched/nohz.h b/include/linux/sched/nohz.h
index 1abe91ff6e4a2..6d67e9a5af6bb 100644
--- a/include/linux/sched/nohz.h
+++ b/include/linux/sched/nohz.h
@@ -15,9 +15,11 @@ static inline void nohz_balance_enter_idle(int cpu) { }
#ifdef CONFIG_NO_HZ_COMMON
void calc_load_nohz_start(void);
+void calc_load_nohz_remote(struct rq *rq);
void calc_load_nohz_stop(void);
#else
static inline void calc_load_nohz_start(void) { }
+static inline void calc_load_nohz_remote(struct rq *rq) { }
static inline void calc_load_nohz_stop(void) { }
#endif /* CONFIG_NO_HZ_COMMON */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ee3685385a6a3..1f4f8e850cab7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3676,6 +3676,7 @@ static void sched_tick_remote(struct work_struct *work)
if (cpu_is_offline(cpu))
goto out_unlock;
+ curr = rq->curr;
update_rq_clock(rq);
if (!is_idle_task(curr)) {
@@ -3688,10 +3689,11 @@ static void sched_tick_remote(struct work_struct *work)
}
curr->sched_class->task_tick(rq, curr, 0);
+ calc_load_nohz_remote(rq);
out_unlock:
rq_unlock_irq(rq, &rf);
-
out_requeue:
+
/*
* Run the remote tick once per second (1Hz). This arbitrary
* frequency is large enough to avoid overload but short enough
diff --git a/kernel/sched/loadavg.c b/kernel/sched/loadavg.c
index 28a516575c181..de22da666ac73 100644
--- a/kernel/sched/loadavg.c
+++ b/kernel/sched/loadavg.c
@@ -231,16 +231,11 @@ static inline int calc_load_read_idx(void)
return calc_load_idx & 1;
}
-void calc_load_nohz_start(void)
+static void calc_load_nohz_fold(struct rq *rq)
{
- struct rq *this_rq = this_rq();
long delta;
- /*
- * We're going into NO_HZ mode, if there's any pending delta, fold it
- * into the pending NO_HZ delta.
- */
- delta = calc_load_fold_active(this_rq, 0);
+ delta = calc_load_fold_active(rq, 0);
if (delta) {
int idx = calc_load_write_idx();
@@ -248,6 +243,24 @@ void calc_load_nohz_start(void)
}
}
+void calc_load_nohz_start(void)
+{
+ /*
+ * We're going into NO_HZ mode, if there's any pending delta, fold it
+ * into the pending NO_HZ delta.
+ */
+ calc_load_nohz_fold(this_rq());
+}
+
+/*
+ * Keep track of the load for NOHZ_FULL, must be called between
+ * calc_load_nohz_{start,stop}().
+ */
+void calc_load_nohz_remote(struct rq *rq)
+{
+ calc_load_nohz_fold(rq);
+}
+
void calc_load_nohz_stop(void)
{
struct rq *this_rq = this_rq();
@@ -268,7 +281,7 @@ void calc_load_nohz_stop(void)
this_rq->calc_load_update += LOAD_FREQ;
}
-static long calc_load_nohz_fold(void)
+static long calc_load_nohz_read(void)
{
int idx = calc_load_read_idx();
long delta = 0;
@@ -323,7 +336,7 @@ static void calc_global_nohz(void)
}
#else /* !CONFIG_NO_HZ_COMMON */
-static inline long calc_load_nohz_fold(void) { return 0; }
+static inline long calc_load_nohz_read(void) { return 0; }
static inline void calc_global_nohz(void) { }
#endif /* CONFIG_NO_HZ_COMMON */
@@ -346,7 +359,7 @@ void calc_global_load(unsigned long ticks)
/*
* Fold the 'old' NO_HZ-delta to include all NO_HZ CPUs.
*/
- delta = calc_load_nohz_fold();
+ delta = calc_load_nohz_read();
if (delta)
atomic_long_add(delta, &calc_load_tasks);
--
2.20.1
next prev parent reply other threads:[~2020-02-23 2:38 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-23 2:20 [PATCH AUTOSEL 5.5 01/58] ipmi:ssif: Handle a possible NULL pointer reference Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 02/58] drm/msm: Set dma maximum segment size for mdss Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 03/58] sched/core: Don't skip remote tick for idle CPUs Sasha Levin
2020-02-23 2:20 ` Sasha Levin [this message]
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 05/58] sched/fair: Prevent unlimited runtime on throttled group Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 06/58] dax: pass NOWAIT flag to iomap_apply Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 07/58] mac80211: consider more elements in parsing CRC Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 08/58] cfg80211: check wiphy driver existence for drvinfo report Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 09/58] io_uring: flush overflowed CQ events in the io_uring_poll() Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 10/58] s390/zcrypt: fix card and queue total counter wrap Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 11/58] qmi_wwan: re-add DW5821e pre-production variant Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 12/58] qmi_wwan: unconditionally reject 2 ep interfaces Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 13/58] NFSv4: Fix races between open and dentry revalidation Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 14/58] perf/smmuv3: Use platform_get_irq_optional() for wired interrupt Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 15/58] arm/ftrace: Fix BE text poking Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 16/58] sched/psi: Fix OOB write when writing 0 bytes to PSI files Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 17/58] perf/x86/intel: Add Elkhart Lake support Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 18/58] perf/x86/cstate: Add Tremont support Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 19/58] perf/x86/msr: " Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 20/58] ceph: do not execute direct write in parallel if O_APPEND is specified Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 21/58] ARM: dts: sti: fixup sound frame-inversion for stihxxx-b2120.dtsi Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 22/58] drm/amd/display: Do not set optimized_require to false after plane disable Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 23/58] RDMA/siw: Remove unwanted WARN_ON in siw_cm_llp_data_ready() Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 24/58] drm/amd/display: Check engine is not NULL before acquiring Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 25/58] drm/amd/display: Limit minimum DPPCLK to 100MHz Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 26/58] drm/amd/display: Add initialitions for PLL2 clock source Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 27/58] amdgpu: Prevent build errors regarding soft/hard-float FP ABI tags Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 28/58] soc/tegra: fuse: Fix build with Tegra194 configuration Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 29/58] i40e: Fix the conditional for i40e_vc_validate_vqs_bitmaps Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 30/58] net: ena: fix potential crash when rxfh key is NULL Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 31/58] net: ena: fix uses of round_jiffies() Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 32/58] net: ena: add missing ethtool TX timestamping indication Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 33/58] net: ena: fix incorrect default RSS key Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 34/58] net: ena: rss: do not allocate key when not supported Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 35/58] net: ena: rss: fix failure to get indirection table Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 36/58] net: ena: rss: store hash function as values and not bits Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 37/58] net: ena: fix incorrectly saving queue numbers when setting RSS indirection table Sasha Levin
2020-02-23 2:20 ` [PATCH AUTOSEL 5.5 38/58] net: ena: fix corruption of dev_idx_to_host_tbl Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 39/58] net: ena: ethtool: use correct value for crc32 hash Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 40/58] net: ena: ena-com.c: prevent NULL pointer dereference Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 41/58] enic: prevent waking up stopped tx queues over watchdog reset Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 42/58] ice: Fix switch between FW and SW LLDP Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 43/58] ice: Don't allow same value for Rx tail to be written twice Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 44/58] ice: fix and consolidate logging of NVM/firmware version information Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 45/58] ice: update Unit Load Status bitmask to check after reset Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 46/58] ice: Use ice_pf_to_dev Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 47/58] ice: Use correct netif error function Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 48/58] io-wq: don't call kXalloc_node() with non-online node Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 49/58] cifs: Fix mode output in debugging statements Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 50/58] bcache: ignore pending signals when creating gc and allocator thread Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 51/58] cfg80211: add missing policy for NL80211_ATTR_STATUS_CODE Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 52/58] mac80211: fix wrong 160/80+80 MHz setting Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 53/58] net: hns3: add management table after IMP reset Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 54/58] net: hns3: fix VF bandwidth does not take effect in some case Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 55/58] net: hns3: fix a copying IPv6 address error in hclge_fd_get_flow_tuples() Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 56/58] nvme/tcp: fix bug on double requeue when send fails Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 57/58] nvme: prevent warning triggered by nvme_stop_keep_alive Sasha Levin
2020-02-23 2:21 ` [PATCH AUTOSEL 5.5 58/58] nvme/pci: move cqe check after device shutdown Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200223022119.707-4-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=stable@vger.kernel.org \
--cc=swood@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).