* [PATCH 0/10] Reduce system disruption due to kswapd V3
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Big change is again related to proportional reclaim.
Changelog since V2
o Preserve ratio properly for proportional scanning (kamezawa)
Changelog since V1
o Rename ZONE_DIRTY to ZONE_TAIL_LRU_DIRTY (andi)
o Reformat comment in shrink_page_list (andi)
o Clarify some comments (dhillf)
o Rework how the proportional scanning is preserved
o Add PageReclaim check before kswapd starts writeback
o Reset sc.nr_reclaimed on every full zone scan
Kswapd and page reclaim behaviour has been screwy in one way or the other
for a long time. Very broadly speaking it worked in the far past because
machines were limited in memory so it did not have that many pages to scan
and it stalled congestion_wait() frequently to prevent it going completely
nuts. In recent times it has behaved very unsatisfactorily with some of
the problems compounded by the removal of stall logic and the introduction
of transparent hugepage support with high-order reclaims.
There are many variations of bugs that are rooted in this area. One example
is reports of a large copy operations or backup causing the machine to
grind to a halt or applications pushed to swap. Sometimes in low memory
situations a large percentage of memory suddenly gets reclaimed. In other
cases an application starts and kswapd hits 100% CPU usage for prolonged
periods of time and so on. There is now talk of introducing features like
an extra free kbytes tunable to work around aspects of the problem instead
of trying to deal with it. It's compounded by the problem that it can be
very workload and machine specific.
This series aims at addressing some of the worst of these problems without
attempting to fundmentally alter how page reclaim works.
Patches 1-2 limits the number of pages kswapd reclaims while still obeying
the anon/file proportion of the LRUs it should be scanning.
Patches 3-4 control how and when kswapd raises its scanning priority and
deletes the scanning restart logic which is tricky to follow.
Patch 5 notes that it is too easy for kswapd to reach priority 0 when
scanning and then reclaim the world. Down with that sort of thing.
Patch 6 notes that kswapd starts writeback based on scanning priority which
is not necessarily related to dirty pages. It will have kswapd
writeback pages if a number of unqueued dirty pages have been
recently encountered at the tail of the LRU.
Patch 7 notes that sometimes kswapd should stall waiting on IO to complete
to reduce LRU churn and the likelihood that it'll reclaim young
clean pages or push applications to swap. It will cause kswapd
to block on IO if it detects that pages being reclaimed under
writeback are recycling through the LRU before the IO completes.
Patch 8 shrinks slab just once per priority scanned or if a zone is otherwise
unreclaimable to avoid hammering slab when kswapd has to skip a
large number of pages.
Patches 9-10 are cosmetic but balance_pgdat() might be easier to follow.
This was tested using memcached+memcachetest while some background IO
was in progress as implemented by the parallel IO tests implement in MM
Tests. memcachetest benchmarks how many operations/second memcached can
service and it is run multiple times. It starts with no background IO and
then re-runs the test with larger amounts of IO in the background to roughly
simulate a large copy in progress. The expectation is that the IO should
have little or no impact on memcachetest which is running entirely in memory.
3.9.0-rc6 3.9.0-rc6
vanilla lessdisrupt-v3r6
Ops memcachetest-0M 10868.00 ( 0.00%) 10932.00 ( 0.59%)
Ops memcachetest-749M 10976.00 ( 0.00%) 10986.00 ( 0.09%)
Ops memcachetest-2498M 3406.00 ( 0.00%) 10871.00 (219.17%)
Ops memcachetest-4246M 2402.00 ( 0.00%) 10936.00 (355.29%)
Ops io-duration-0M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops io-duration-749M 15.00 ( 0.00%) 9.00 ( 40.00%)
Ops io-duration-2498M 107.00 ( 0.00%) 27.00 ( 74.77%)
Ops io-duration-4246M 193.00 ( 0.00%) 47.00 ( 75.65%)
Ops swaptotal-0M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops swaptotal-749M 155965.00 ( 0.00%) 25.00 ( 99.98%)
Ops swaptotal-2498M 335917.00 ( 0.00%) 287.00 ( 99.91%)
Ops swaptotal-4246M 463021.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-0M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-749M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-2498M 139128.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-4246M 156276.00 ( 0.00%) 0.00 ( 0.00%)
Ops minorfaults-0M 1677257.00 ( 0.00%) 1642376.00 ( 2.08%)
Ops minorfaults-749M 1819566.00 ( 0.00%) 1572243.00 ( 13.59%)
Ops minorfaults-2498M 1842140.00 ( 0.00%) 1652508.00 ( 10.29%)
Ops minorfaults-4246M 1796116.00 ( 0.00%) 1651464.00 ( 8.05%)
Ops majorfaults-0M 6.00 ( 0.00%) 6.00 ( 0.00%)
Ops majorfaults-749M 55.00 ( 0.00%) 49.00 ( 10.91%)
Ops majorfaults-2498M 20936.00 ( 0.00%) 110.00 ( 99.47%)
Ops majorfaults-4246M 22487.00 ( 0.00%) 185.00 ( 99.18%)
Note how the vanilla kernels performance collapses when there is enough IO
taking place in the background. This drop in performance is part of users
complain of when they start backups. Note how the swapin and major fault
figures indicate that processes were being pushed to swap prematurely. With
the series applied, there is no noticable performance drop and while there
is still some swap activity, it's tiny.
3.9.0-rc6 3.9.0-rc6
vanillalessdisrupt-v3r6
Page Ins 1281068 89224
Page Outs 15697620 11478616
Swap Ins 295654 0
Swap Outs 659499 312
Direct pages scanned 0 78668
Kswapd pages scanned 7166977 4416457
Kswapd pages reclaimed 1185518 1051751
Direct pages reclaimed 0 72993
Kswapd efficiency 16% 23%
Kswapd velocity 5558.640 3420.614
Direct efficiency 100% 92%
Direct velocity 0.000 60.930
Percentage direct scans 0% 1%
Page writes by reclaim 2044715 2922251
Page writes file 1385216 2921939
Page writes anon 659499 312
Page reclaim immediate 4040 218
Page rescued immediate 0 0
Slabs scanned 35456 26624
Direct inode steals 0 0
Kswapd inode steals 19898 1420
Kswapd skipped wait 0 0
THP fault alloc 11 51
THP collapse alloc 574 609
THP splits 9 6
THP fault fallback 0 0
THP collapse fail 0 0
Compaction stalls 0 0
Compaction success 0 0
Compaction failures 0 0
Page migrate success 0 0
Page migrate failure 0 0
Compaction pages isolated 0 0
Compaction migrate scanned 0 0
Compaction free scanned 0 0
Compaction cost 0 0
NUMA PTE updates 0 0
NUMA hint faults 0 0
NUMA hint local faults 0 0
NUMA pages migrated 0 0
AutoNUMA cost 0 0
Note that kswapd efficiency is slightly improved. Unfortunately, also note
that there is a small amount of direct reclaim due to kswapd no longer reclaiming
the world. Using ftrace it would appear that the direct reclaim stalls are mostly
harmless with the vast bulk of the stalls incurred by dd
2 gzip-3111
5 memcachetest-12607
26 tclsh-3109
67 tee-3110
89 flush-8:0-286
2055 dd-12795
There is a risk that kswapd not reclaiming the world may mean that it
stays awake balancing zones, does not stall on the appropriate events
and continually scans pages it cannot reclaim consuming CPU. This will
be visible as continued high CPU usage but in my own tests I only saw a
single spike lasting less than a second and I did not observe any problems
related to reclaim while running the series on my desktop.
include/linux/mmzone.h | 17 ++
mm/vmscan.c | 461 ++++++++++++++++++++++++++++++-------------------
2 files changed, 305 insertions(+), 173 deletions(-)
--
1.8.1.4
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH 0/10] Reduce system disruption due to kswapd V3
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Big change is again related to proportional reclaim.
Changelog since V2
o Preserve ratio properly for proportional scanning (kamezawa)
Changelog since V1
o Rename ZONE_DIRTY to ZONE_TAIL_LRU_DIRTY (andi)
o Reformat comment in shrink_page_list (andi)
o Clarify some comments (dhillf)
o Rework how the proportional scanning is preserved
o Add PageReclaim check before kswapd starts writeback
o Reset sc.nr_reclaimed on every full zone scan
Kswapd and page reclaim behaviour has been screwy in one way or the other
for a long time. Very broadly speaking it worked in the far past because
machines were limited in memory so it did not have that many pages to scan
and it stalled congestion_wait() frequently to prevent it going completely
nuts. In recent times it has behaved very unsatisfactorily with some of
the problems compounded by the removal of stall logic and the introduction
of transparent hugepage support with high-order reclaims.
There are many variations of bugs that are rooted in this area. One example
is reports of a large copy operations or backup causing the machine to
grind to a halt or applications pushed to swap. Sometimes in low memory
situations a large percentage of memory suddenly gets reclaimed. In other
cases an application starts and kswapd hits 100% CPU usage for prolonged
periods of time and so on. There is now talk of introducing features like
an extra free kbytes tunable to work around aspects of the problem instead
of trying to deal with it. It's compounded by the problem that it can be
very workload and machine specific.
This series aims at addressing some of the worst of these problems without
attempting to fundmentally alter how page reclaim works.
Patches 1-2 limits the number of pages kswapd reclaims while still obeying
the anon/file proportion of the LRUs it should be scanning.
Patches 3-4 control how and when kswapd raises its scanning priority and
deletes the scanning restart logic which is tricky to follow.
Patch 5 notes that it is too easy for kswapd to reach priority 0 when
scanning and then reclaim the world. Down with that sort of thing.
Patch 6 notes that kswapd starts writeback based on scanning priority which
is not necessarily related to dirty pages. It will have kswapd
writeback pages if a number of unqueued dirty pages have been
recently encountered at the tail of the LRU.
Patch 7 notes that sometimes kswapd should stall waiting on IO to complete
to reduce LRU churn and the likelihood that it'll reclaim young
clean pages or push applications to swap. It will cause kswapd
to block on IO if it detects that pages being reclaimed under
writeback are recycling through the LRU before the IO completes.
Patch 8 shrinks slab just once per priority scanned or if a zone is otherwise
unreclaimable to avoid hammering slab when kswapd has to skip a
large number of pages.
Patches 9-10 are cosmetic but balance_pgdat() might be easier to follow.
This was tested using memcached+memcachetest while some background IO
was in progress as implemented by the parallel IO tests implement in MM
Tests. memcachetest benchmarks how many operations/second memcached can
service and it is run multiple times. It starts with no background IO and
then re-runs the test with larger amounts of IO in the background to roughly
simulate a large copy in progress. The expectation is that the IO should
have little or no impact on memcachetest which is running entirely in memory.
3.9.0-rc6 3.9.0-rc6
vanilla lessdisrupt-v3r6
Ops memcachetest-0M 10868.00 ( 0.00%) 10932.00 ( 0.59%)
Ops memcachetest-749M 10976.00 ( 0.00%) 10986.00 ( 0.09%)
Ops memcachetest-2498M 3406.00 ( 0.00%) 10871.00 (219.17%)
Ops memcachetest-4246M 2402.00 ( 0.00%) 10936.00 (355.29%)
Ops io-duration-0M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops io-duration-749M 15.00 ( 0.00%) 9.00 ( 40.00%)
Ops io-duration-2498M 107.00 ( 0.00%) 27.00 ( 74.77%)
Ops io-duration-4246M 193.00 ( 0.00%) 47.00 ( 75.65%)
Ops swaptotal-0M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops swaptotal-749M 155965.00 ( 0.00%) 25.00 ( 99.98%)
Ops swaptotal-2498M 335917.00 ( 0.00%) 287.00 ( 99.91%)
Ops swaptotal-4246M 463021.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-0M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-749M 0.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-2498M 139128.00 ( 0.00%) 0.00 ( 0.00%)
Ops swapin-4246M 156276.00 ( 0.00%) 0.00 ( 0.00%)
Ops minorfaults-0M 1677257.00 ( 0.00%) 1642376.00 ( 2.08%)
Ops minorfaults-749M 1819566.00 ( 0.00%) 1572243.00 ( 13.59%)
Ops minorfaults-2498M 1842140.00 ( 0.00%) 1652508.00 ( 10.29%)
Ops minorfaults-4246M 1796116.00 ( 0.00%) 1651464.00 ( 8.05%)
Ops majorfaults-0M 6.00 ( 0.00%) 6.00 ( 0.00%)
Ops majorfaults-749M 55.00 ( 0.00%) 49.00 ( 10.91%)
Ops majorfaults-2498M 20936.00 ( 0.00%) 110.00 ( 99.47%)
Ops majorfaults-4246M 22487.00 ( 0.00%) 185.00 ( 99.18%)
Note how the vanilla kernels performance collapses when there is enough IO
taking place in the background. This drop in performance is part of users
complain of when they start backups. Note how the swapin and major fault
figures indicate that processes were being pushed to swap prematurely. With
the series applied, there is no noticable performance drop and while there
is still some swap activity, it's tiny.
3.9.0-rc6 3.9.0-rc6
vanillalessdisrupt-v3r6
Page Ins 1281068 89224
Page Outs 15697620 11478616
Swap Ins 295654 0
Swap Outs 659499 312
Direct pages scanned 0 78668
Kswapd pages scanned 7166977 4416457
Kswapd pages reclaimed 1185518 1051751
Direct pages reclaimed 0 72993
Kswapd efficiency 16% 23%
Kswapd velocity 5558.640 3420.614
Direct efficiency 100% 92%
Direct velocity 0.000 60.930
Percentage direct scans 0% 1%
Page writes by reclaim 2044715 2922251
Page writes file 1385216 2921939
Page writes anon 659499 312
Page reclaim immediate 4040 218
Page rescued immediate 0 0
Slabs scanned 35456 26624
Direct inode steals 0 0
Kswapd inode steals 19898 1420
Kswapd skipped wait 0 0
THP fault alloc 11 51
THP collapse alloc 574 609
THP splits 9 6
THP fault fallback 0 0
THP collapse fail 0 0
Compaction stalls 0 0
Compaction success 0 0
Compaction failures 0 0
Page migrate success 0 0
Page migrate failure 0 0
Compaction pages isolated 0 0
Compaction migrate scanned 0 0
Compaction free scanned 0 0
Compaction cost 0 0
NUMA PTE updates 0 0
NUMA hint faults 0 0
NUMA hint local faults 0 0
NUMA pages migrated 0 0
AutoNUMA cost 0 0
Note that kswapd efficiency is slightly improved. Unfortunately, also note
that there is a small amount of direct reclaim due to kswapd no longer reclaiming
the world. Using ftrace it would appear that the direct reclaim stalls are mostly
harmless with the vast bulk of the stalls incurred by dd
2 gzip-3111
5 memcachetest-12607
26 tclsh-3109
67 tee-3110
89 flush-8:0-286
2055 dd-12795
There is a risk that kswapd not reclaiming the world may mean that it
stays awake balancing zones, does not stall on the appropriate events
and continually scans pages it cannot reclaim consuming CPU. This will
be visible as continued high CPU usage but in my own tests I only saw a
single spike lasting less than a second and I did not observe any problems
related to reclaim while running the series on my desktop.
include/linux/mmzone.h | 17 ++
mm/vmscan.c | 461 ++++++++++++++++++++++++++++++-------------------
2 files changed, 305 insertions(+), 173 deletions(-)
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH 01/10] mm: vmscan: Limit the number of pages kswapd reclaims at each priority
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
The number of pages kswapd can reclaim is bound by the number of pages it
scans which is related to the size of the zone and the scanning priority. In
many cases the priority remains low because it's reset every SWAP_CLUSTER_MAX
reclaimed pages but in the event kswapd scans a large number of pages it
cannot reclaim, it will raise the priority and potentially discard a large
percentage of the zone as sc->nr_to_reclaim is ULONG_MAX. The user-visible
effect is a reclaim "spike" where a large percentage of memory is suddenly
freed. It would be bad enough if this was just unused memory but because
of how anon/file pages are balanced it is possible that applications get
pushed to swap unnecessarily.
This patch limits the number of pages kswapd will reclaim to the high
watermark. Reclaim will still overshoot due to it not being a hard limit as
shrink_lruvec() will ignore the sc.nr_to_reclaim at DEF_PRIORITY but it
prevents kswapd reclaiming the world at higher priorities. The number of
pages it reclaims is not adjusted for high-order allocations as kswapd will
reclaim excessively if it is to balance zones for high-order allocations.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Rik van Riel <riel@redhat.com>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
---
mm/vmscan.c | 53 +++++++++++++++++++++++++++++------------------------
1 file changed, 29 insertions(+), 24 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 88c5fed..4835a7a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2593,6 +2593,32 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
}
/*
+ * kswapd shrinks the zone by the number of pages required to reach
+ * the high watermark.
+ */
+static void kswapd_shrink_zone(struct zone *zone,
+ struct scan_control *sc,
+ unsigned long lru_pages)
+{
+ unsigned long nr_slab;
+ struct reclaim_state *reclaim_state = current->reclaim_state;
+ struct shrink_control shrink = {
+ .gfp_mask = sc->gfp_mask,
+ };
+
+ /* Reclaim above the high watermark. */
+ sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+ shrink_zone(zone, sc);
+
+ reclaim_state->reclaimed_slab = 0;
+ nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
+ sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+
+ if (nr_slab == 0 && !zone_reclaimable(zone))
+ zone->all_unreclaimable = 1;
+}
+
+/*
* For kswapd, balance_pgdat() will work across all this node's zones until
* they are all at high_wmark_pages(zone).
*
@@ -2619,27 +2645,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
bool pgdat_is_balanced = false;
int i;
int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */
- unsigned long total_scanned;
- struct reclaim_state *reclaim_state = current->reclaim_state;
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
.may_unmap = 1,
.may_swap = 1,
- /*
- * kswapd doesn't want to be bailed out while reclaim. because
- * we want to put equal scanning pressure on each zone.
- */
- .nr_to_reclaim = ULONG_MAX,
.order = order,
.target_mem_cgroup = NULL,
};
- struct shrink_control shrink = {
- .gfp_mask = sc.gfp_mask,
- };
loop_again:
- total_scanned = 0;
sc.priority = DEF_PRIORITY;
sc.nr_reclaimed = 0;
sc.may_writepage = !laptop_mode;
@@ -2710,7 +2725,7 @@ loop_again:
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int nr_slab, testorder;
+ int testorder;
unsigned long balance_gap;
if (!populated_zone(zone))
@@ -2730,7 +2745,6 @@ loop_again:
order, sc.gfp_mask,
&nr_soft_scanned);
sc.nr_reclaimed += nr_soft_reclaimed;
- total_scanned += nr_soft_scanned;
/*
* We put equal pressure on every zone, unless
@@ -2759,17 +2773,8 @@ loop_again:
if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
!zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- shrink_zone(zone, &sc);
-
- reclaim_state->reclaimed_slab = 0;
- nr_slab = shrink_slab(&shrink, sc.nr_scanned, lru_pages);
- sc.nr_reclaimed += reclaim_state->reclaimed_slab;
- total_scanned += sc.nr_scanned;
-
- if (nr_slab == 0 && !zone_reclaimable(zone))
- zone->all_unreclaimable = 1;
- }
+ balance_gap, end_zone))
+ kswapd_shrink_zone(zone, &sc, lru_pages);
/*
* If we're getting trouble reclaiming, start doing
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 01/10] mm: vmscan: Limit the number of pages kswapd reclaims at each priority
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
The number of pages kswapd can reclaim is bound by the number of pages it
scans which is related to the size of the zone and the scanning priority. In
many cases the priority remains low because it's reset every SWAP_CLUSTER_MAX
reclaimed pages but in the event kswapd scans a large number of pages it
cannot reclaim, it will raise the priority and potentially discard a large
percentage of the zone as sc->nr_to_reclaim is ULONG_MAX. The user-visible
effect is a reclaim "spike" where a large percentage of memory is suddenly
freed. It would be bad enough if this was just unused memory but because
of how anon/file pages are balanced it is possible that applications get
pushed to swap unnecessarily.
This patch limits the number of pages kswapd will reclaim to the high
watermark. Reclaim will still overshoot due to it not being a hard limit as
shrink_lruvec() will ignore the sc.nr_to_reclaim at DEF_PRIORITY but it
prevents kswapd reclaiming the world at higher priorities. The number of
pages it reclaims is not adjusted for high-order allocations as kswapd will
reclaim excessively if it is to balance zones for high-order allocations.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Rik van Riel <riel@redhat.com>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
---
mm/vmscan.c | 53 +++++++++++++++++++++++++++++------------------------
1 file changed, 29 insertions(+), 24 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 88c5fed..4835a7a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2593,6 +2593,32 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
}
/*
+ * kswapd shrinks the zone by the number of pages required to reach
+ * the high watermark.
+ */
+static void kswapd_shrink_zone(struct zone *zone,
+ struct scan_control *sc,
+ unsigned long lru_pages)
+{
+ unsigned long nr_slab;
+ struct reclaim_state *reclaim_state = current->reclaim_state;
+ struct shrink_control shrink = {
+ .gfp_mask = sc->gfp_mask,
+ };
+
+ /* Reclaim above the high watermark. */
+ sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+ shrink_zone(zone, sc);
+
+ reclaim_state->reclaimed_slab = 0;
+ nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
+ sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+
+ if (nr_slab == 0 && !zone_reclaimable(zone))
+ zone->all_unreclaimable = 1;
+}
+
+/*
* For kswapd, balance_pgdat() will work across all this node's zones until
* they are all at high_wmark_pages(zone).
*
@@ -2619,27 +2645,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
bool pgdat_is_balanced = false;
int i;
int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */
- unsigned long total_scanned;
- struct reclaim_state *reclaim_state = current->reclaim_state;
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
.may_unmap = 1,
.may_swap = 1,
- /*
- * kswapd doesn't want to be bailed out while reclaim. because
- * we want to put equal scanning pressure on each zone.
- */
- .nr_to_reclaim = ULONG_MAX,
.order = order,
.target_mem_cgroup = NULL,
};
- struct shrink_control shrink = {
- .gfp_mask = sc.gfp_mask,
- };
loop_again:
- total_scanned = 0;
sc.priority = DEF_PRIORITY;
sc.nr_reclaimed = 0;
sc.may_writepage = !laptop_mode;
@@ -2710,7 +2725,7 @@ loop_again:
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int nr_slab, testorder;
+ int testorder;
unsigned long balance_gap;
if (!populated_zone(zone))
@@ -2730,7 +2745,6 @@ loop_again:
order, sc.gfp_mask,
&nr_soft_scanned);
sc.nr_reclaimed += nr_soft_reclaimed;
- total_scanned += nr_soft_scanned;
/*
* We put equal pressure on every zone, unless
@@ -2759,17 +2773,8 @@ loop_again:
if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
!zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- shrink_zone(zone, &sc);
-
- reclaim_state->reclaimed_slab = 0;
- nr_slab = shrink_slab(&shrink, sc.nr_scanned, lru_pages);
- sc.nr_reclaimed += reclaim_state->reclaimed_slab;
- total_scanned += sc.nr_scanned;
-
- if (nr_slab == 0 && !zone_reclaimable(zone))
- zone->all_unreclaimable = 1;
- }
+ balance_gap, end_zone))
+ kswapd_shrink_zone(zone, &sc, lru_pages);
/*
* If we're getting trouble reclaiming, start doing
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 02/10] mm: vmscan: Obey proportional scanning requirements for kswapd
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Simplistically, the anon and file LRU lists are scanned proportionally
depending on the value of vm.swappiness although there are other factors
taken into account by get_scan_count(). The patch "mm: vmscan: Limit
the number of pages kswapd reclaims" limits the number of pages kswapd
reclaims but it breaks this proportional scanning and may evenly shrink
anon/file LRUs regardless of vm.swappiness.
This patch preserves the proportional scanning and reclaim. It does mean
that kswapd will reclaim more than requested but the number of pages will
be related to the high watermark.
[mhocko@suse.cz: Correct proportional reclaim for memcg and simplify]
[kamezawa.hiroyu@jp.fujitsu.com: Recalculate scan based on target]
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
---
mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 56 insertions(+), 8 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 4835a7a..a6bca2c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1821,17 +1821,24 @@ out:
static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
{
unsigned long nr[NR_LRU_LISTS];
+ unsigned long targets[NR_LRU_LISTS];
unsigned long nr_to_scan;
enum lru_list lru;
unsigned long nr_reclaimed = 0;
unsigned long nr_to_reclaim = sc->nr_to_reclaim;
struct blk_plug plug;
+ bool scan_adjusted = false;
get_scan_count(lruvec, sc, nr);
+ /* Record the original scan target for proportional adjustments later */
+ memcpy(targets, nr, sizeof(nr));
+
blk_start_plug(&plug);
while (nr[LRU_INACTIVE_ANON] || nr[LRU_ACTIVE_FILE] ||
nr[LRU_INACTIVE_FILE]) {
+ unsigned long nr_anon, nr_file, percentage;
+
for_each_evictable_lru(lru) {
if (nr[lru]) {
nr_to_scan = min(nr[lru], SWAP_CLUSTER_MAX);
@@ -1841,17 +1848,58 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
lruvec, sc);
}
}
+
+ if (nr_reclaimed < nr_to_reclaim || scan_adjusted)
+ continue;
+
/*
- * On large memory systems, scan >> priority can become
- * really large. This is fine for the starting priority;
- * we want to put equal scanning pressure on each zone.
- * However, if the VM has a harder time of freeing pages,
- * with multiple processes reclaiming pages, the total
- * freeing target can get unreasonably large.
+ * For global direct reclaim, reclaim only the number of pages
+ * requested. Less care is taken to scan proportionally as it
+ * is more important to minimise direct reclaim stall latency
+ * than it is to properly age the LRU lists.
*/
- if (nr_reclaimed >= nr_to_reclaim &&
- sc->priority < DEF_PRIORITY)
+ if (global_reclaim(sc) && !current_is_kswapd())
break;
+
+ /*
+ * For kswapd and memcg, reclaim at least the number of pages
+ * requested. Ensure that the anon and file LRUs shrink
+ * proportionally what was requested by get_scan_count(). We
+ * stop reclaiming one LRU and reduce the amount scanning
+ * proportional to the original scan target.
+ */
+ nr_file = nr[LRU_INACTIVE_FILE] + nr[LRU_ACTIVE_FILE];
+ nr_anon = nr[LRU_INACTIVE_ANON] + nr[LRU_ACTIVE_ANON];
+
+ if (nr_file > nr_anon) {
+ unsigned long scan_target = targets[LRU_INACTIVE_ANON] +
+ targets[LRU_ACTIVE_ANON] + 1;
+ lru = LRU_BASE;
+ percentage = nr_anon * 100 / scan_target;
+ } else {
+ unsigned long scan_target = targets[LRU_INACTIVE_FILE] +
+ targets[LRU_ACTIVE_FILE] + 1;
+ lru = LRU_FILE;
+ percentage = nr_file * 100 / scan_target;
+ }
+
+ /* Stop scanning the smaller of the LRU */
+ nr[lru] = 0;
+ nr[lru + LRU_ACTIVE] = 0;
+
+ /*
+ * Recalculate the other LRU scan count based on its original
+ * scan target and the percentage scanning already complete
+ */
+ lru = (lru == LRU_FILE) ? LRU_BASE : LRU_FILE;
+ nr[lru] = targets[lru] * (100 - percentage) / 100;
+ nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
+
+ lru += LRU_ACTIVE;
+ nr[lru] = targets[lru] * (100 - percentage) / 100;
+ nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
+
+ scan_adjusted = true;
}
blk_finish_plug(&plug);
sc->nr_reclaimed += nr_reclaimed;
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 02/10] mm: vmscan: Obey proportional scanning requirements for kswapd
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Simplistically, the anon and file LRU lists are scanned proportionally
depending on the value of vm.swappiness although there are other factors
taken into account by get_scan_count(). The patch "mm: vmscan: Limit
the number of pages kswapd reclaims" limits the number of pages kswapd
reclaims but it breaks this proportional scanning and may evenly shrink
anon/file LRUs regardless of vm.swappiness.
This patch preserves the proportional scanning and reclaim. It does mean
that kswapd will reclaim more than requested but the number of pages will
be related to the high watermark.
[mhocko@suse.cz: Correct proportional reclaim for memcg and simplify]
[kamezawa.hiroyu@jp.fujitsu.com: Recalculate scan based on target]
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
---
mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 56 insertions(+), 8 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 4835a7a..a6bca2c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1821,17 +1821,24 @@ out:
static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
{
unsigned long nr[NR_LRU_LISTS];
+ unsigned long targets[NR_LRU_LISTS];
unsigned long nr_to_scan;
enum lru_list lru;
unsigned long nr_reclaimed = 0;
unsigned long nr_to_reclaim = sc->nr_to_reclaim;
struct blk_plug plug;
+ bool scan_adjusted = false;
get_scan_count(lruvec, sc, nr);
+ /* Record the original scan target for proportional adjustments later */
+ memcpy(targets, nr, sizeof(nr));
+
blk_start_plug(&plug);
while (nr[LRU_INACTIVE_ANON] || nr[LRU_ACTIVE_FILE] ||
nr[LRU_INACTIVE_FILE]) {
+ unsigned long nr_anon, nr_file, percentage;
+
for_each_evictable_lru(lru) {
if (nr[lru]) {
nr_to_scan = min(nr[lru], SWAP_CLUSTER_MAX);
@@ -1841,17 +1848,58 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
lruvec, sc);
}
}
+
+ if (nr_reclaimed < nr_to_reclaim || scan_adjusted)
+ continue;
+
/*
- * On large memory systems, scan >> priority can become
- * really large. This is fine for the starting priority;
- * we want to put equal scanning pressure on each zone.
- * However, if the VM has a harder time of freeing pages,
- * with multiple processes reclaiming pages, the total
- * freeing target can get unreasonably large.
+ * For global direct reclaim, reclaim only the number of pages
+ * requested. Less care is taken to scan proportionally as it
+ * is more important to minimise direct reclaim stall latency
+ * than it is to properly age the LRU lists.
*/
- if (nr_reclaimed >= nr_to_reclaim &&
- sc->priority < DEF_PRIORITY)
+ if (global_reclaim(sc) && !current_is_kswapd())
break;
+
+ /*
+ * For kswapd and memcg, reclaim at least the number of pages
+ * requested. Ensure that the anon and file LRUs shrink
+ * proportionally what was requested by get_scan_count(). We
+ * stop reclaiming one LRU and reduce the amount scanning
+ * proportional to the original scan target.
+ */
+ nr_file = nr[LRU_INACTIVE_FILE] + nr[LRU_ACTIVE_FILE];
+ nr_anon = nr[LRU_INACTIVE_ANON] + nr[LRU_ACTIVE_ANON];
+
+ if (nr_file > nr_anon) {
+ unsigned long scan_target = targets[LRU_INACTIVE_ANON] +
+ targets[LRU_ACTIVE_ANON] + 1;
+ lru = LRU_BASE;
+ percentage = nr_anon * 100 / scan_target;
+ } else {
+ unsigned long scan_target = targets[LRU_INACTIVE_FILE] +
+ targets[LRU_ACTIVE_FILE] + 1;
+ lru = LRU_FILE;
+ percentage = nr_file * 100 / scan_target;
+ }
+
+ /* Stop scanning the smaller of the LRU */
+ nr[lru] = 0;
+ nr[lru + LRU_ACTIVE] = 0;
+
+ /*
+ * Recalculate the other LRU scan count based on its original
+ * scan target and the percentage scanning already complete
+ */
+ lru = (lru == LRU_FILE) ? LRU_BASE : LRU_FILE;
+ nr[lru] = targets[lru] * (100 - percentage) / 100;
+ nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
+
+ lru += LRU_ACTIVE;
+ nr[lru] = targets[lru] * (100 - percentage) / 100;
+ nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
+
+ scan_adjusted = true;
}
blk_finish_plug(&plug);
sc->nr_reclaimed += nr_reclaimed;
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 03/10] mm: vmscan: Flatten kswapd priority loop
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
kswapd stops raising the scanning priority when at least SWAP_CLUSTER_MAX
pages have been reclaimed or the pgdat is considered balanced. It then
rechecks if it needs to restart at DEF_PRIORITY and whether high-order
reclaim needs to be reset. This is not wrong per-se but it is confusing
to follow and forcing kswapd to stay at DEF_PRIORITY may require several
restarts before it has scanned enough pages to meet the high watermark even
at 100% efficiency. This patch irons out the logic a bit by controlling
when priority is raised and removing the "goto loop_again".
This patch has kswapd raise the scanning priority until it is scanning
enough pages that it could meet the high watermark in one shrink of the
LRU lists if it is able to reclaim at 100% efficiency. It will not raise
the scanning prioirty higher unless it is failing to reclaim any pages.
To avoid infinite looping for high-order allocation requests kswapd will
not reclaim for high-order allocations when it has reclaimed at least
twice the number of pages as the allocation request.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 86 +++++++++++++++++++++++++++++--------------------------------
1 file changed, 41 insertions(+), 45 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a6bca2c..f979a67 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2643,8 +2643,12 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
/*
* kswapd shrinks the zone by the number of pages required to reach
* the high watermark.
+ *
+ * Returns true if kswapd scanned at least the requested number of pages to
+ * reclaim. This is used to determine if the scanning priority needs to be
+ * raised.
*/
-static void kswapd_shrink_zone(struct zone *zone,
+static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
unsigned long lru_pages)
{
@@ -2664,6 +2668,8 @@ static void kswapd_shrink_zone(struct zone *zone,
if (nr_slab == 0 && !zone_reclaimable(zone))
zone->all_unreclaimable = 1;
+
+ return sc->nr_scanned >= sc->nr_to_reclaim;
}
/*
@@ -2690,26 +2696,26 @@ static void kswapd_shrink_zone(struct zone *zone,
static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
int *classzone_idx)
{
- bool pgdat_is_balanced = false;
int i;
int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
+ .priority = DEF_PRIORITY,
.may_unmap = 1,
.may_swap = 1,
+ .may_writepage = !laptop_mode,
.order = order,
.target_mem_cgroup = NULL,
};
-loop_again:
- sc.priority = DEF_PRIORITY;
- sc.nr_reclaimed = 0;
- sc.may_writepage = !laptop_mode;
count_vm_event(PAGEOUTRUN);
do {
unsigned long lru_pages = 0;
+ bool raise_priority = true;
+
+ sc.nr_reclaimed = 0;
/*
* Scan in the highmem->dma direction for the highest
@@ -2751,10 +2757,8 @@ loop_again:
}
}
- if (i < 0) {
- pgdat_is_balanced = true;
+ if (i < 0)
goto out;
- }
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
@@ -2821,8 +2825,16 @@ loop_again:
if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
!zone_balanced(zone, testorder,
- balance_gap, end_zone))
- kswapd_shrink_zone(zone, &sc, lru_pages);
+ balance_gap, end_zone)) {
+ /*
+ * There should be no need to raise the
+ * scanning priority if enough pages are
+ * already being scanned that high
+ * watermark would be met at 100% efficiency.
+ */
+ if (kswapd_shrink_zone(zone, &sc, lru_pages))
+ raise_priority = false;
+ }
/*
* If we're getting trouble reclaiming, start doing
@@ -2857,46 +2869,29 @@ loop_again:
pfmemalloc_watermark_ok(pgdat))
wake_up(&pgdat->pfmemalloc_wait);
- if (pgdat_balanced(pgdat, order, *classzone_idx)) {
- pgdat_is_balanced = true;
- break; /* kswapd: all done */
- }
-
/*
- * We do this so kswapd doesn't build up large priorities for
- * example when it is freeing in parallel with allocators. It
- * matches the direct reclaim path behaviour in terms of impact
- * on zone->*_priority.
+ * Fragmentation may mean that the system cannot be rebalanced
+ * for high-order allocations in all zones. If twice the
+ * allocation size has been reclaimed and the zones are still
+ * not balanced then recheck the watermarks at order-0 to
+ * prevent kswapd reclaiming excessively. Assume that a
+ * process requested a high-order can direct reclaim/compact.
*/
- if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
- break;
- } while (--sc.priority >= 0);
-
-out:
- if (!pgdat_is_balanced) {
- cond_resched();
+ if (order && sc.nr_reclaimed >= 2UL << order)
+ order = sc.order = 0;
- try_to_freeze();
+ /* Check if kswapd should be suspending */
+ if (try_to_freeze() || kthread_should_stop())
+ break;
/*
- * Fragmentation may mean that the system cannot be
- * rebalanced for high-order allocations in all zones.
- * At this point, if nr_reclaimed < SWAP_CLUSTER_MAX,
- * it means the zones have been fully scanned and are still
- * not balanced. For high-order allocations, there is
- * little point trying all over again as kswapd may
- * infinite loop.
- *
- * Instead, recheck all watermarks at order-0 as they
- * are the most important. If watermarks are ok, kswapd will go
- * back to sleep. High-order users can still perform direct
- * reclaim if they wish.
+ * Raise priority if scanning rate is too low or there was no
+ * progress in reclaiming pages
*/
- if (sc.nr_reclaimed < SWAP_CLUSTER_MAX)
- order = sc.order = 0;
-
- goto loop_again;
- }
+ if (raise_priority || !sc.nr_reclaimed)
+ sc.priority--;
+ } while (sc.priority >= 0 &&
+ !pgdat_balanced(pgdat, order, *classzone_idx));
/*
* If kswapd was reclaiming at a higher order, it has the option of
@@ -2925,6 +2920,7 @@ out:
compact_pgdat(pgdat, order);
}
+out:
/*
* Return the order we were reclaiming at so prepare_kswapd_sleep()
* makes a decision on the order we were last reclaiming at. However,
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 03/10] mm: vmscan: Flatten kswapd priority loop
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
kswapd stops raising the scanning priority when at least SWAP_CLUSTER_MAX
pages have been reclaimed or the pgdat is considered balanced. It then
rechecks if it needs to restart at DEF_PRIORITY and whether high-order
reclaim needs to be reset. This is not wrong per-se but it is confusing
to follow and forcing kswapd to stay at DEF_PRIORITY may require several
restarts before it has scanned enough pages to meet the high watermark even
at 100% efficiency. This patch irons out the logic a bit by controlling
when priority is raised and removing the "goto loop_again".
This patch has kswapd raise the scanning priority until it is scanning
enough pages that it could meet the high watermark in one shrink of the
LRU lists if it is able to reclaim at 100% efficiency. It will not raise
the scanning prioirty higher unless it is failing to reclaim any pages.
To avoid infinite looping for high-order allocation requests kswapd will
not reclaim for high-order allocations when it has reclaimed at least
twice the number of pages as the allocation request.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 86 +++++++++++++++++++++++++++++--------------------------------
1 file changed, 41 insertions(+), 45 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a6bca2c..f979a67 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2643,8 +2643,12 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
/*
* kswapd shrinks the zone by the number of pages required to reach
* the high watermark.
+ *
+ * Returns true if kswapd scanned at least the requested number of pages to
+ * reclaim. This is used to determine if the scanning priority needs to be
+ * raised.
*/
-static void kswapd_shrink_zone(struct zone *zone,
+static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
unsigned long lru_pages)
{
@@ -2664,6 +2668,8 @@ static void kswapd_shrink_zone(struct zone *zone,
if (nr_slab == 0 && !zone_reclaimable(zone))
zone->all_unreclaimable = 1;
+
+ return sc->nr_scanned >= sc->nr_to_reclaim;
}
/*
@@ -2690,26 +2696,26 @@ static void kswapd_shrink_zone(struct zone *zone,
static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
int *classzone_idx)
{
- bool pgdat_is_balanced = false;
int i;
int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
+ .priority = DEF_PRIORITY,
.may_unmap = 1,
.may_swap = 1,
+ .may_writepage = !laptop_mode,
.order = order,
.target_mem_cgroup = NULL,
};
-loop_again:
- sc.priority = DEF_PRIORITY;
- sc.nr_reclaimed = 0;
- sc.may_writepage = !laptop_mode;
count_vm_event(PAGEOUTRUN);
do {
unsigned long lru_pages = 0;
+ bool raise_priority = true;
+
+ sc.nr_reclaimed = 0;
/*
* Scan in the highmem->dma direction for the highest
@@ -2751,10 +2757,8 @@ loop_again:
}
}
- if (i < 0) {
- pgdat_is_balanced = true;
+ if (i < 0)
goto out;
- }
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
@@ -2821,8 +2825,16 @@ loop_again:
if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
!zone_balanced(zone, testorder,
- balance_gap, end_zone))
- kswapd_shrink_zone(zone, &sc, lru_pages);
+ balance_gap, end_zone)) {
+ /*
+ * There should be no need to raise the
+ * scanning priority if enough pages are
+ * already being scanned that high
+ * watermark would be met at 100% efficiency.
+ */
+ if (kswapd_shrink_zone(zone, &sc, lru_pages))
+ raise_priority = false;
+ }
/*
* If we're getting trouble reclaiming, start doing
@@ -2857,46 +2869,29 @@ loop_again:
pfmemalloc_watermark_ok(pgdat))
wake_up(&pgdat->pfmemalloc_wait);
- if (pgdat_balanced(pgdat, order, *classzone_idx)) {
- pgdat_is_balanced = true;
- break; /* kswapd: all done */
- }
-
/*
- * We do this so kswapd doesn't build up large priorities for
- * example when it is freeing in parallel with allocators. It
- * matches the direct reclaim path behaviour in terms of impact
- * on zone->*_priority.
+ * Fragmentation may mean that the system cannot be rebalanced
+ * for high-order allocations in all zones. If twice the
+ * allocation size has been reclaimed and the zones are still
+ * not balanced then recheck the watermarks at order-0 to
+ * prevent kswapd reclaiming excessively. Assume that a
+ * process requested a high-order can direct reclaim/compact.
*/
- if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
- break;
- } while (--sc.priority >= 0);
-
-out:
- if (!pgdat_is_balanced) {
- cond_resched();
+ if (order && sc.nr_reclaimed >= 2UL << order)
+ order = sc.order = 0;
- try_to_freeze();
+ /* Check if kswapd should be suspending */
+ if (try_to_freeze() || kthread_should_stop())
+ break;
/*
- * Fragmentation may mean that the system cannot be
- * rebalanced for high-order allocations in all zones.
- * At this point, if nr_reclaimed < SWAP_CLUSTER_MAX,
- * it means the zones have been fully scanned and are still
- * not balanced. For high-order allocations, there is
- * little point trying all over again as kswapd may
- * infinite loop.
- *
- * Instead, recheck all watermarks at order-0 as they
- * are the most important. If watermarks are ok, kswapd will go
- * back to sleep. High-order users can still perform direct
- * reclaim if they wish.
+ * Raise priority if scanning rate is too low or there was no
+ * progress in reclaiming pages
*/
- if (sc.nr_reclaimed < SWAP_CLUSTER_MAX)
- order = sc.order = 0;
-
- goto loop_again;
- }
+ if (raise_priority || !sc.nr_reclaimed)
+ sc.priority--;
+ } while (sc.priority >= 0 &&
+ !pgdat_balanced(pgdat, order, *classzone_idx));
/*
* If kswapd was reclaiming at a higher order, it has the option of
@@ -2925,6 +2920,7 @@ out:
compact_pgdat(pgdat, order);
}
+out:
/*
* Return the order we were reclaiming at so prepare_kswapd_sleep()
* makes a decision on the order we were last reclaiming at. However,
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 04/10] mm: vmscan: Decide whether to compact the pgdat based on reclaim progress
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
In the past, kswapd makes a decision on whether to compact memory after the
pgdat was considered balanced. This more or less worked but it is late to
make such a decision and does not fit well now that kswapd makes a decision
whether to exit the zone scanning loop depending on reclaim progress.
This patch will compact a pgdat if at least the requested number of pages
were reclaimed from unbalanced zones for a given priority. If any zone is
currently balanced, kswapd will not call compaction as it is expected the
necessary pages are already available.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 59 ++++++++++++++++++++++++++++++-----------------------------
1 file changed, 30 insertions(+), 29 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f979a67..25d89af 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2650,7 +2650,8 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
*/
static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
- unsigned long lru_pages)
+ unsigned long lru_pages,
+ unsigned long *nr_attempted)
{
unsigned long nr_slab;
struct reclaim_state *reclaim_state = current->reclaim_state;
@@ -2666,6 +2667,9 @@ static bool kswapd_shrink_zone(struct zone *zone,
nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+ /* Account for the number of pages attempted to reclaim */
+ *nr_attempted += sc->nr_to_reclaim;
+
if (nr_slab == 0 && !zone_reclaimable(zone))
zone->all_unreclaimable = 1;
@@ -2713,7 +2717,9 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
do {
unsigned long lru_pages = 0;
+ unsigned long nr_attempted = 0;
bool raise_priority = true;
+ bool pgdat_needs_compaction = (order > 0);
sc.nr_reclaimed = 0;
@@ -2763,7 +2769,21 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
+ if (!populated_zone(zone))
+ continue;
+
lru_pages += zone_reclaimable_pages(zone);
+
+ /*
+ * If any zone is currently balanced then kswapd will
+ * not call compaction as it is expected that the
+ * necessary pages are already available.
+ */
+ if (pgdat_needs_compaction &&
+ zone_watermark_ok(zone, order,
+ low_wmark_pages(zone),
+ *classzone_idx, 0))
+ pgdat_needs_compaction = false;
}
/*
@@ -2832,7 +2852,8 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* already being scanned that high
* watermark would be met at 100% efficiency.
*/
- if (kswapd_shrink_zone(zone, &sc, lru_pages))
+ if (kswapd_shrink_zone(zone, &sc, lru_pages,
+ &nr_attempted))
raise_priority = false;
}
@@ -2885,6 +2906,13 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
break;
/*
+ * Compact if necessary and kswapd is reclaiming at least the
+ * high watermark number of pages as requsted
+ */
+ if (pgdat_needs_compaction && sc.nr_reclaimed > nr_attempted)
+ compact_pgdat(pgdat, order);
+
+ /*
* Raise priority if scanning rate is too low or there was no
* progress in reclaiming pages
*/
@@ -2893,33 +2921,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
} while (sc.priority >= 0 &&
!pgdat_balanced(pgdat, order, *classzone_idx));
- /*
- * If kswapd was reclaiming at a higher order, it has the option of
- * sleeping without all zones being balanced. Before it does, it must
- * ensure that the watermarks for order-0 on *all* zones are met and
- * that the congestion flags are cleared. The congestion flag must
- * be cleared as kswapd is the only mechanism that clears the flag
- * and it is potentially going to sleep here.
- */
- if (order) {
- int zones_need_compaction = 1;
-
- for (i = 0; i <= end_zone; i++) {
- struct zone *zone = pgdat->node_zones + i;
-
- if (!populated_zone(zone))
- continue;
-
- /* Check if the memory needs to be defragmented. */
- if (zone_watermark_ok(zone, order,
- low_wmark_pages(zone), *classzone_idx, 0))
- zones_need_compaction = 0;
- }
-
- if (zones_need_compaction)
- compact_pgdat(pgdat, order);
- }
-
out:
/*
* Return the order we were reclaiming at so prepare_kswapd_sleep()
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 04/10] mm: vmscan: Decide whether to compact the pgdat based on reclaim progress
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
In the past, kswapd makes a decision on whether to compact memory after the
pgdat was considered balanced. This more or less worked but it is late to
make such a decision and does not fit well now that kswapd makes a decision
whether to exit the zone scanning loop depending on reclaim progress.
This patch will compact a pgdat if at least the requested number of pages
were reclaimed from unbalanced zones for a given priority. If any zone is
currently balanced, kswapd will not call compaction as it is expected the
necessary pages are already available.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 59 ++++++++++++++++++++++++++++++-----------------------------
1 file changed, 30 insertions(+), 29 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f979a67..25d89af 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2650,7 +2650,8 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
*/
static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
- unsigned long lru_pages)
+ unsigned long lru_pages,
+ unsigned long *nr_attempted)
{
unsigned long nr_slab;
struct reclaim_state *reclaim_state = current->reclaim_state;
@@ -2666,6 +2667,9 @@ static bool kswapd_shrink_zone(struct zone *zone,
nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+ /* Account for the number of pages attempted to reclaim */
+ *nr_attempted += sc->nr_to_reclaim;
+
if (nr_slab == 0 && !zone_reclaimable(zone))
zone->all_unreclaimable = 1;
@@ -2713,7 +2717,9 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
do {
unsigned long lru_pages = 0;
+ unsigned long nr_attempted = 0;
bool raise_priority = true;
+ bool pgdat_needs_compaction = (order > 0);
sc.nr_reclaimed = 0;
@@ -2763,7 +2769,21 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
+ if (!populated_zone(zone))
+ continue;
+
lru_pages += zone_reclaimable_pages(zone);
+
+ /*
+ * If any zone is currently balanced then kswapd will
+ * not call compaction as it is expected that the
+ * necessary pages are already available.
+ */
+ if (pgdat_needs_compaction &&
+ zone_watermark_ok(zone, order,
+ low_wmark_pages(zone),
+ *classzone_idx, 0))
+ pgdat_needs_compaction = false;
}
/*
@@ -2832,7 +2852,8 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* already being scanned that high
* watermark would be met at 100% efficiency.
*/
- if (kswapd_shrink_zone(zone, &sc, lru_pages))
+ if (kswapd_shrink_zone(zone, &sc, lru_pages,
+ &nr_attempted))
raise_priority = false;
}
@@ -2885,6 +2906,13 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
break;
/*
+ * Compact if necessary and kswapd is reclaiming at least the
+ * high watermark number of pages as requsted
+ */
+ if (pgdat_needs_compaction && sc.nr_reclaimed > nr_attempted)
+ compact_pgdat(pgdat, order);
+
+ /*
* Raise priority if scanning rate is too low or there was no
* progress in reclaiming pages
*/
@@ -2893,33 +2921,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
} while (sc.priority >= 0 &&
!pgdat_balanced(pgdat, order, *classzone_idx));
- /*
- * If kswapd was reclaiming at a higher order, it has the option of
- * sleeping without all zones being balanced. Before it does, it must
- * ensure that the watermarks for order-0 on *all* zones are met and
- * that the congestion flags are cleared. The congestion flag must
- * be cleared as kswapd is the only mechanism that clears the flag
- * and it is potentially going to sleep here.
- */
- if (order) {
- int zones_need_compaction = 1;
-
- for (i = 0; i <= end_zone; i++) {
- struct zone *zone = pgdat->node_zones + i;
-
- if (!populated_zone(zone))
- continue;
-
- /* Check if the memory needs to be defragmented. */
- if (zone_watermark_ok(zone, order,
- low_wmark_pages(zone), *classzone_idx, 0))
- zones_need_compaction = 0;
- }
-
- if (zones_need_compaction)
- compact_pgdat(pgdat, order);
- }
-
out:
/*
* Return the order we were reclaiming at so prepare_kswapd_sleep()
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 05/10] mm: vmscan: Do not allow kswapd to scan at maximum priority
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Page reclaim at priority 0 will scan the entire LRU as priority 0 is
considered to be a near OOM condition. Kswapd can reach priority 0 quite
easily if it is encountering a large number of pages it cannot reclaim
such as pages under writeback. When this happens, kswapd reclaims very
aggressively even though there may be no real risk of allocation failure
or OOM.
This patch prevents kswapd reaching priority 0 and trying to reclaim
the world. Direct reclaimers will still reach priority 0 in the event
of an OOM situation.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
---
mm/vmscan.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 25d89af..bc4c2a7 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2918,7 +2918,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
if (raise_priority || !sc.nr_reclaimed)
sc.priority--;
- } while (sc.priority >= 0 &&
+ } while (sc.priority >= 1 &&
!pgdat_balanced(pgdat, order, *classzone_idx));
out:
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 05/10] mm: vmscan: Do not allow kswapd to scan at maximum priority
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Page reclaim at priority 0 will scan the entire LRU as priority 0 is
considered to be a near OOM condition. Kswapd can reach priority 0 quite
easily if it is encountering a large number of pages it cannot reclaim
such as pages under writeback. When this happens, kswapd reclaims very
aggressively even though there may be no real risk of allocation failure
or OOM.
This patch prevents kswapd reaching priority 0 and trying to reclaim
the world. Direct reclaimers will still reach priority 0 in the event
of an OOM situation.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
---
mm/vmscan.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 25d89af..bc4c2a7 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2918,7 +2918,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
if (raise_priority || !sc.nr_reclaimed)
sc.priority--;
- } while (sc.priority >= 0 &&
+ } while (sc.priority >= 1 &&
!pgdat_balanced(pgdat, order, *classzone_idx));
out:
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 06/10] mm: vmscan: Have kswapd writeback pages based on dirty pages encountered, not priority
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Currently kswapd queues dirty pages for writeback if scanning at an elevated
priority but the priority kswapd scans at is not related to the number
of unqueued dirty encountered. Since commit "mm: vmscan: Flatten kswapd
priority loop", the priority is related to the size of the LRU and the
zone watermark which is no indication as to whether kswapd should write
pages or not.
This patch tracks if an excessive number of unqueued dirty pages are being
encountered at the end of the LRU. If so, it indicates that dirty pages
are being recycled before flusher threads can clean them and flags the
zone so that kswapd will start writing pages until the zone is balanced.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/mmzone.h | 9 +++++++++
mm/vmscan.c | 31 +++++++++++++++++++++++++------
2 files changed, 34 insertions(+), 6 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index c74092e..ecf0c7d 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -495,6 +495,10 @@ typedef enum {
ZONE_CONGESTED, /* zone has many dirty pages backed by
* a congested BDI
*/
+ ZONE_TAIL_LRU_DIRTY, /* reclaim scanning has recently found
+ * many dirty file pages at the tail
+ * of the LRU.
+ */
} zone_flags_t;
static inline void zone_set_flag(struct zone *zone, zone_flags_t flag)
@@ -517,6 +521,11 @@ static inline int zone_is_reclaim_congested(const struct zone *zone)
return test_bit(ZONE_CONGESTED, &zone->flags);
}
+static inline int zone_is_reclaim_dirty(const struct zone *zone)
+{
+ return test_bit(ZONE_TAIL_LRU_DIRTY, &zone->flags);
+}
+
static inline int zone_is_reclaim_locked(const struct zone *zone)
{
return test_bit(ZONE_RECLAIM_LOCKED, &zone->flags);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index bc4c2a7..22e8ca9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -675,13 +675,14 @@ static unsigned long shrink_page_list(struct list_head *page_list,
struct zone *zone,
struct scan_control *sc,
enum ttu_flags ttu_flags,
- unsigned long *ret_nr_dirty,
+ unsigned long *ret_nr_unqueued_dirty,
unsigned long *ret_nr_writeback,
bool force_reclaim)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
int pgactivate = 0;
+ unsigned long nr_unqueued_dirty = 0;
unsigned long nr_dirty = 0;
unsigned long nr_congested = 0;
unsigned long nr_reclaimed = 0;
@@ -807,14 +808,17 @@ static unsigned long shrink_page_list(struct list_head *page_list,
if (PageDirty(page)) {
nr_dirty++;
+ if (!PageWriteback(page))
+ nr_unqueued_dirty++;
+
/*
* Only kswapd can writeback filesystem pages to
- * avoid risk of stack overflow but do not writeback
- * unless under significant pressure.
+ * avoid risk of stack overflow but only writeback
+ * if many dirty pages have been encountered.
*/
if (page_is_file_cache(page) &&
(!current_is_kswapd() ||
- sc->priority >= DEF_PRIORITY - 2)) {
+ !zone_is_reclaim_dirty(zone))) {
/*
* Immediately reclaim when written back.
* Similar in principal to deactivate_page()
@@ -959,7 +963,7 @@ keep:
list_splice(&ret_pages, page_list);
count_vm_events(PGACTIVATE, pgactivate);
mem_cgroup_uncharge_end();
- *ret_nr_dirty += nr_dirty;
+ *ret_nr_unqueued_dirty += nr_unqueued_dirty;
*ret_nr_writeback += nr_writeback;
return nr_reclaimed;
}
@@ -1372,6 +1376,15 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
(nr_taken >> (DEF_PRIORITY - sc->priority)))
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+ /*
+ * Similarly, if many dirty pages are encountered that are not
+ * currently being written then flag that kswapd should start
+ * writing back pages.
+ */
+ if (global_reclaim(sc) && nr_dirty &&
+ nr_dirty >= (nr_taken >> (DEF_PRIORITY - sc->priority)))
+ zone_set_flag(zone, ZONE_TAIL_LRU_DIRTY);
+
trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
zone_idx(zone),
nr_scanned, nr_reclaimed,
@@ -2758,8 +2771,12 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
end_zone = i;
break;
} else {
- /* If balanced, clear the congested flag */
+ /*
+ * If balanced, clear the dirty and congested
+ * flags
+ */
zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
}
}
@@ -2877,8 +2894,10 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* possible there are dirty pages backed by
* congested BDIs but as pressure is relieved,
* speculatively avoid congestion waits
+ * or writing pages from kswapd context.
*/
zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
}
/*
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 06/10] mm: vmscan: Have kswapd writeback pages based on dirty pages encountered, not priority
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Currently kswapd queues dirty pages for writeback if scanning at an elevated
priority but the priority kswapd scans at is not related to the number
of unqueued dirty encountered. Since commit "mm: vmscan: Flatten kswapd
priority loop", the priority is related to the size of the LRU and the
zone watermark which is no indication as to whether kswapd should write
pages or not.
This patch tracks if an excessive number of unqueued dirty pages are being
encountered at the end of the LRU. If so, it indicates that dirty pages
are being recycled before flusher threads can clean them and flags the
zone so that kswapd will start writing pages until the zone is balanced.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
include/linux/mmzone.h | 9 +++++++++
mm/vmscan.c | 31 +++++++++++++++++++++++++------
2 files changed, 34 insertions(+), 6 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index c74092e..ecf0c7d 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -495,6 +495,10 @@ typedef enum {
ZONE_CONGESTED, /* zone has many dirty pages backed by
* a congested BDI
*/
+ ZONE_TAIL_LRU_DIRTY, /* reclaim scanning has recently found
+ * many dirty file pages at the tail
+ * of the LRU.
+ */
} zone_flags_t;
static inline void zone_set_flag(struct zone *zone, zone_flags_t flag)
@@ -517,6 +521,11 @@ static inline int zone_is_reclaim_congested(const struct zone *zone)
return test_bit(ZONE_CONGESTED, &zone->flags);
}
+static inline int zone_is_reclaim_dirty(const struct zone *zone)
+{
+ return test_bit(ZONE_TAIL_LRU_DIRTY, &zone->flags);
+}
+
static inline int zone_is_reclaim_locked(const struct zone *zone)
{
return test_bit(ZONE_RECLAIM_LOCKED, &zone->flags);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index bc4c2a7..22e8ca9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -675,13 +675,14 @@ static unsigned long shrink_page_list(struct list_head *page_list,
struct zone *zone,
struct scan_control *sc,
enum ttu_flags ttu_flags,
- unsigned long *ret_nr_dirty,
+ unsigned long *ret_nr_unqueued_dirty,
unsigned long *ret_nr_writeback,
bool force_reclaim)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
int pgactivate = 0;
+ unsigned long nr_unqueued_dirty = 0;
unsigned long nr_dirty = 0;
unsigned long nr_congested = 0;
unsigned long nr_reclaimed = 0;
@@ -807,14 +808,17 @@ static unsigned long shrink_page_list(struct list_head *page_list,
if (PageDirty(page)) {
nr_dirty++;
+ if (!PageWriteback(page))
+ nr_unqueued_dirty++;
+
/*
* Only kswapd can writeback filesystem pages to
- * avoid risk of stack overflow but do not writeback
- * unless under significant pressure.
+ * avoid risk of stack overflow but only writeback
+ * if many dirty pages have been encountered.
*/
if (page_is_file_cache(page) &&
(!current_is_kswapd() ||
- sc->priority >= DEF_PRIORITY - 2)) {
+ !zone_is_reclaim_dirty(zone))) {
/*
* Immediately reclaim when written back.
* Similar in principal to deactivate_page()
@@ -959,7 +963,7 @@ keep:
list_splice(&ret_pages, page_list);
count_vm_events(PGACTIVATE, pgactivate);
mem_cgroup_uncharge_end();
- *ret_nr_dirty += nr_dirty;
+ *ret_nr_unqueued_dirty += nr_unqueued_dirty;
*ret_nr_writeback += nr_writeback;
return nr_reclaimed;
}
@@ -1372,6 +1376,15 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
(nr_taken >> (DEF_PRIORITY - sc->priority)))
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+ /*
+ * Similarly, if many dirty pages are encountered that are not
+ * currently being written then flag that kswapd should start
+ * writing back pages.
+ */
+ if (global_reclaim(sc) && nr_dirty &&
+ nr_dirty >= (nr_taken >> (DEF_PRIORITY - sc->priority)))
+ zone_set_flag(zone, ZONE_TAIL_LRU_DIRTY);
+
trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
zone_idx(zone),
nr_scanned, nr_reclaimed,
@@ -2758,8 +2771,12 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
end_zone = i;
break;
} else {
- /* If balanced, clear the congested flag */
+ /*
+ * If balanced, clear the dirty and congested
+ * flags
+ */
zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
}
}
@@ -2877,8 +2894,10 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* possible there are dirty pages backed by
* congested BDIs but as pressure is relieved,
* speculatively avoid congestion waits
+ * or writing pages from kswapd context.
*/
zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
}
/*
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 07/10] mm: vmscan: Block kswapd if it is encountering pages under writeback
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Historically, kswapd used to congestion_wait() at higher priorities if it
was not making forward progress. This made no sense as the failure to make
progress could be completely independent of IO. It was later replaced by
wait_iff_congested() and removed entirely by commit 258401a6 (mm: don't
wait on congested zones in balance_pgdat()) as it was duplicating logic
in shrink_inactive_list().
This is problematic. If kswapd encounters many pages under writeback and
it continues to scan until it reaches the high watermark then it will
quickly skip over the pages under writeback and reclaim clean young
pages or push applications out to swap.
The use of wait_iff_congested() is not suited to kswapd as it will only
stall if the underlying BDI is really congested or a direct reclaimer was
unable to write to the underlying BDI. kswapd bypasses the BDI congestion
as it sets PF_SWAPWRITE but even if this was taken into account then it
would cause direct reclaimers to stall on writeback which is not desirable.
This patch sets a ZONE_WRITEBACK flag if direct reclaim or kswapd is
encountering too many pages under writeback. If this flag is set and
kswapd encounters a PageReclaim page under writeback then it'll assume
that the LRU lists are being recycled too quickly before IO can complete
and block waiting for some IO to complete.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
---
include/linux/mmzone.h | 8 ++++++
mm/vmscan.c | 78 ++++++++++++++++++++++++++++++++++++--------------
2 files changed, 64 insertions(+), 22 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecf0c7d..264e203 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -499,6 +499,9 @@ typedef enum {
* many dirty file pages at the tail
* of the LRU.
*/
+ ZONE_WRITEBACK, /* reclaim scanning has recently found
+ * many pages under writeback
+ */
} zone_flags_t;
static inline void zone_set_flag(struct zone *zone, zone_flags_t flag)
@@ -526,6 +529,11 @@ static inline int zone_is_reclaim_dirty(const struct zone *zone)
return test_bit(ZONE_TAIL_LRU_DIRTY, &zone->flags);
}
+static inline int zone_is_reclaim_writeback(const struct zone *zone)
+{
+ return test_bit(ZONE_WRITEBACK, &zone->flags);
+}
+
static inline int zone_is_reclaim_locked(const struct zone *zone)
{
return test_bit(ZONE_RECLAIM_LOCKED, &zone->flags);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 22e8ca9..a20f2a9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -723,25 +723,51 @@ static unsigned long shrink_page_list(struct list_head *page_list,
may_enter_fs = (sc->gfp_mask & __GFP_FS) ||
(PageSwapCache(page) && (sc->gfp_mask & __GFP_IO));
+ /*
+ * If a page at the tail of the LRU is under writeback, there
+ * are three cases to consider.
+ *
+ * 1) If reclaim is encountering an excessive number of pages
+ * under writeback and this page is both under writeback and
+ * PageReclaim then it indicates that pages are being queued
+ * for IO but are being recycled through the LRU before the
+ * IO can complete. In this case, wait on the IO to complete
+ * and then clear the ZONE_WRITEBACK flag to recheck if the
+ * condition exists.
+ *
+ * 2) Global reclaim encounters a page, memcg encounters a
+ * page that is not marked for immediate reclaim or
+ * the caller does not have __GFP_IO. In this case mark
+ * the page for immediate reclaim and continue scanning.
+ *
+ * __GFP_IO is checked because a loop driver thread might
+ * enter reclaim, and deadlock if it waits on a page for
+ * which it is needed to do the write (loop masks off
+ * __GFP_IO|__GFP_FS for this reason); but more thought
+ * would probably show more reasons.
+ *
+ * Don't require __GFP_FS, since we're not going into the
+ * FS, just waiting on its writeback completion. Worryingly,
+ * ext4 gfs2 and xfs allocate pages with
+ * grab_cache_page_write_begin(,,AOP_FLAG_NOFS), so testing
+ * may_enter_fs here is liable to OOM on them.
+ *
+ * 3) memcg encounters a page that is not already marked
+ * PageReclaim. memcg does not have any dirty pages
+ * throttling so we could easily OOM just because too many
+ * pages are in writeback and there is nothing else to
+ * reclaim. Wait for the writeback to complete.
+ */
if (PageWriteback(page)) {
- /*
- * memcg doesn't have any dirty pages throttling so we
- * could easily OOM just because too many pages are in
- * writeback and there is nothing else to reclaim.
- *
- * Check __GFP_IO, certainly because a loop driver
- * thread might enter reclaim, and deadlock if it waits
- * on a page for which it is needed to do the write
- * (loop masks off __GFP_IO|__GFP_FS for this reason);
- * but more thought would probably show more reasons.
- *
- * Don't require __GFP_FS, since we're not going into
- * the FS, just waiting on its writeback completion.
- * Worryingly, ext4 gfs2 and xfs allocate pages with
- * grab_cache_page_write_begin(,,AOP_FLAG_NOFS), so
- * testing may_enter_fs here is liable to OOM on them.
- */
- if (global_reclaim(sc) ||
+ /* Case 1 above */
+ if (current_is_kswapd() &&
+ PageReclaim(page) &&
+ zone_is_reclaim_writeback(zone)) {
+ wait_on_page_writeback(page);
+ zone_clear_flag(zone, ZONE_WRITEBACK);
+
+ /* Case 2 above */
+ } else if (global_reclaim(sc) ||
!PageReclaim(page) || !(sc->gfp_mask & __GFP_IO)) {
/*
* This is slightly racy - end_page_writeback()
@@ -756,9 +782,13 @@ static unsigned long shrink_page_list(struct list_head *page_list,
*/
SetPageReclaim(page);
nr_writeback++;
+
goto keep_locked;
+
+ /* Case 3 above */
+ } else {
+ wait_on_page_writeback(page);
}
- wait_on_page_writeback(page);
}
if (!force_reclaim)
@@ -1373,8 +1403,10 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
* isolated page is PageWriteback
*/
if (nr_writeback && nr_writeback >=
- (nr_taken >> (DEF_PRIORITY - sc->priority)))
+ (nr_taken >> (DEF_PRIORITY - sc->priority))) {
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+ zone_set_flag(zone, ZONE_WRITEBACK);
+ }
/*
* Similarly, if many dirty pages are encountered that are not
@@ -2658,8 +2690,8 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* the high watermark.
*
* Returns true if kswapd scanned at least the requested number of pages to
- * reclaim. This is used to determine if the scanning priority needs to be
- * raised.
+ * reclaim or if the lack of progress was due to pages under writeback.
+ * This is used to determine if the scanning priority needs to be raised.
*/
static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
@@ -2686,6 +2718,8 @@ static bool kswapd_shrink_zone(struct zone *zone,
if (nr_slab == 0 && !zone_reclaimable(zone))
zone->all_unreclaimable = 1;
+ zone_clear_flag(zone, ZONE_WRITEBACK);
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 07/10] mm: vmscan: Block kswapd if it is encountering pages under writeback
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Historically, kswapd used to congestion_wait() at higher priorities if it
was not making forward progress. This made no sense as the failure to make
progress could be completely independent of IO. It was later replaced by
wait_iff_congested() and removed entirely by commit 258401a6 (mm: don't
wait on congested zones in balance_pgdat()) as it was duplicating logic
in shrink_inactive_list().
This is problematic. If kswapd encounters many pages under writeback and
it continues to scan until it reaches the high watermark then it will
quickly skip over the pages under writeback and reclaim clean young
pages or push applications out to swap.
The use of wait_iff_congested() is not suited to kswapd as it will only
stall if the underlying BDI is really congested or a direct reclaimer was
unable to write to the underlying BDI. kswapd bypasses the BDI congestion
as it sets PF_SWAPWRITE but even if this was taken into account then it
would cause direct reclaimers to stall on writeback which is not desirable.
This patch sets a ZONE_WRITEBACK flag if direct reclaim or kswapd is
encountering too many pages under writeback. If this flag is set and
kswapd encounters a PageReclaim page under writeback then it'll assume
that the LRU lists are being recycled too quickly before IO can complete
and block waiting for some IO to complete.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
---
include/linux/mmzone.h | 8 ++++++
mm/vmscan.c | 78 ++++++++++++++++++++++++++++++++++++--------------
2 files changed, 64 insertions(+), 22 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecf0c7d..264e203 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -499,6 +499,9 @@ typedef enum {
* many dirty file pages at the tail
* of the LRU.
*/
+ ZONE_WRITEBACK, /* reclaim scanning has recently found
+ * many pages under writeback
+ */
} zone_flags_t;
static inline void zone_set_flag(struct zone *zone, zone_flags_t flag)
@@ -526,6 +529,11 @@ static inline int zone_is_reclaim_dirty(const struct zone *zone)
return test_bit(ZONE_TAIL_LRU_DIRTY, &zone->flags);
}
+static inline int zone_is_reclaim_writeback(const struct zone *zone)
+{
+ return test_bit(ZONE_WRITEBACK, &zone->flags);
+}
+
static inline int zone_is_reclaim_locked(const struct zone *zone)
{
return test_bit(ZONE_RECLAIM_LOCKED, &zone->flags);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 22e8ca9..a20f2a9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -723,25 +723,51 @@ static unsigned long shrink_page_list(struct list_head *page_list,
may_enter_fs = (sc->gfp_mask & __GFP_FS) ||
(PageSwapCache(page) && (sc->gfp_mask & __GFP_IO));
+ /*
+ * If a page at the tail of the LRU is under writeback, there
+ * are three cases to consider.
+ *
+ * 1) If reclaim is encountering an excessive number of pages
+ * under writeback and this page is both under writeback and
+ * PageReclaim then it indicates that pages are being queued
+ * for IO but are being recycled through the LRU before the
+ * IO can complete. In this case, wait on the IO to complete
+ * and then clear the ZONE_WRITEBACK flag to recheck if the
+ * condition exists.
+ *
+ * 2) Global reclaim encounters a page, memcg encounters a
+ * page that is not marked for immediate reclaim or
+ * the caller does not have __GFP_IO. In this case mark
+ * the page for immediate reclaim and continue scanning.
+ *
+ * __GFP_IO is checked because a loop driver thread might
+ * enter reclaim, and deadlock if it waits on a page for
+ * which it is needed to do the write (loop masks off
+ * __GFP_IO|__GFP_FS for this reason); but more thought
+ * would probably show more reasons.
+ *
+ * Don't require __GFP_FS, since we're not going into the
+ * FS, just waiting on its writeback completion. Worryingly,
+ * ext4 gfs2 and xfs allocate pages with
+ * grab_cache_page_write_begin(,,AOP_FLAG_NOFS), so testing
+ * may_enter_fs here is liable to OOM on them.
+ *
+ * 3) memcg encounters a page that is not already marked
+ * PageReclaim. memcg does not have any dirty pages
+ * throttling so we could easily OOM just because too many
+ * pages are in writeback and there is nothing else to
+ * reclaim. Wait for the writeback to complete.
+ */
if (PageWriteback(page)) {
- /*
- * memcg doesn't have any dirty pages throttling so we
- * could easily OOM just because too many pages are in
- * writeback and there is nothing else to reclaim.
- *
- * Check __GFP_IO, certainly because a loop driver
- * thread might enter reclaim, and deadlock if it waits
- * on a page for which it is needed to do the write
- * (loop masks off __GFP_IO|__GFP_FS for this reason);
- * but more thought would probably show more reasons.
- *
- * Don't require __GFP_FS, since we're not going into
- * the FS, just waiting on its writeback completion.
- * Worryingly, ext4 gfs2 and xfs allocate pages with
- * grab_cache_page_write_begin(,,AOP_FLAG_NOFS), so
- * testing may_enter_fs here is liable to OOM on them.
- */
- if (global_reclaim(sc) ||
+ /* Case 1 above */
+ if (current_is_kswapd() &&
+ PageReclaim(page) &&
+ zone_is_reclaim_writeback(zone)) {
+ wait_on_page_writeback(page);
+ zone_clear_flag(zone, ZONE_WRITEBACK);
+
+ /* Case 2 above */
+ } else if (global_reclaim(sc) ||
!PageReclaim(page) || !(sc->gfp_mask & __GFP_IO)) {
/*
* This is slightly racy - end_page_writeback()
@@ -756,9 +782,13 @@ static unsigned long shrink_page_list(struct list_head *page_list,
*/
SetPageReclaim(page);
nr_writeback++;
+
goto keep_locked;
+
+ /* Case 3 above */
+ } else {
+ wait_on_page_writeback(page);
}
- wait_on_page_writeback(page);
}
if (!force_reclaim)
@@ -1373,8 +1403,10 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
* isolated page is PageWriteback
*/
if (nr_writeback && nr_writeback >=
- (nr_taken >> (DEF_PRIORITY - sc->priority)))
+ (nr_taken >> (DEF_PRIORITY - sc->priority))) {
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+ zone_set_flag(zone, ZONE_WRITEBACK);
+ }
/*
* Similarly, if many dirty pages are encountered that are not
@@ -2658,8 +2690,8 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* the high watermark.
*
* Returns true if kswapd scanned at least the requested number of pages to
- * reclaim. This is used to determine if the scanning priority needs to be
- * raised.
+ * reclaim or if the lack of progress was due to pages under writeback.
+ * This is used to determine if the scanning priority needs to be raised.
*/
static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
@@ -2686,6 +2718,8 @@ static bool kswapd_shrink_zone(struct zone *zone,
if (nr_slab == 0 && !zone_reclaimable(zone))
zone->all_unreclaimable = 1;
+ zone_clear_flag(zone, ZONE_WRITEBACK);
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 08/10] mm: vmscan: Have kswapd shrink slab only once per priority
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
If kswaps fails to make progress but continues to shrink slab then it'll
either discard all of slab or consume CPU uselessly scanning shrinkers.
This patch causes kswapd to only call the shrinkers once per priority.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Rik van Riel <riel@redhat.com>
---
mm/vmscan.c | 28 +++++++++++++++++++++-------
1 file changed, 21 insertions(+), 7 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a20f2a9..0fa588d 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2696,9 +2696,10 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
unsigned long lru_pages,
+ bool shrinking_slab,
unsigned long *nr_attempted)
{
- unsigned long nr_slab;
+ unsigned long nr_slab = 0;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
@@ -2708,9 +2709,15 @@ static bool kswapd_shrink_zone(struct zone *zone,
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
shrink_zone(zone, sc);
- reclaim_state->reclaimed_slab = 0;
- nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
- sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+ /*
+ * Slabs are shrunk for each zone once per priority or if the zone
+ * being balanced is otherwise unreclaimable
+ */
+ if (shrinking_slab || !zone_reclaimable(zone)) {
+ reclaim_state->reclaimed_slab = 0;
+ nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
+ sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+ }
/* Account for the number of pages attempted to reclaim */
*nr_attempted += sc->nr_to_reclaim;
@@ -2751,6 +2758,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
+ bool shrinking_slab = true;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
.priority = DEF_PRIORITY,
@@ -2903,8 +2911,9 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* already being scanned that high
* watermark would be met at 100% efficiency.
*/
- if (kswapd_shrink_zone(zone, &sc, lru_pages,
- &nr_attempted))
+ if (kswapd_shrink_zone(zone, &sc,
+ lru_pages, shrinking_slab,
+ &nr_attempted))
raise_priority = false;
}
@@ -2943,6 +2952,9 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
pfmemalloc_watermark_ok(pgdat))
wake_up(&pgdat->pfmemalloc_wait);
+ /* Only shrink slab once per priority */
+ shrinking_slab = false;
+
/*
* Fragmentation may mean that the system cannot be rebalanced
* for high-order allocations in all zones. If twice the
@@ -2969,8 +2981,10 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* Raise priority if scanning rate is too low or there was no
* progress in reclaiming pages
*/
- if (raise_priority || !sc.nr_reclaimed)
+ if (raise_priority || !sc.nr_reclaimed) {
sc.priority--;
+ shrinking_slab = true;
+ }
} while (sc.priority >= 1 &&
!pgdat_balanced(pgdat, order, *classzone_idx));
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 08/10] mm: vmscan: Have kswapd shrink slab only once per priority
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
If kswaps fails to make progress but continues to shrink slab then it'll
either discard all of slab or consume CPU uselessly scanning shrinkers.
This patch causes kswapd to only call the shrinkers once per priority.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Rik van Riel <riel@redhat.com>
---
mm/vmscan.c | 28 +++++++++++++++++++++-------
1 file changed, 21 insertions(+), 7 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a20f2a9..0fa588d 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2696,9 +2696,10 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
static bool kswapd_shrink_zone(struct zone *zone,
struct scan_control *sc,
unsigned long lru_pages,
+ bool shrinking_slab,
unsigned long *nr_attempted)
{
- unsigned long nr_slab;
+ unsigned long nr_slab = 0;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
@@ -2708,9 +2709,15 @@ static bool kswapd_shrink_zone(struct zone *zone,
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
shrink_zone(zone, sc);
- reclaim_state->reclaimed_slab = 0;
- nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
- sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+ /*
+ * Slabs are shrunk for each zone once per priority or if the zone
+ * being balanced is otherwise unreclaimable
+ */
+ if (shrinking_slab || !zone_reclaimable(zone)) {
+ reclaim_state->reclaimed_slab = 0;
+ nr_slab = shrink_slab(&shrink, sc->nr_scanned, lru_pages);
+ sc->nr_reclaimed += reclaim_state->reclaimed_slab;
+ }
/* Account for the number of pages attempted to reclaim */
*nr_attempted += sc->nr_to_reclaim;
@@ -2751,6 +2758,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
+ bool shrinking_slab = true;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
.priority = DEF_PRIORITY,
@@ -2903,8 +2911,9 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* already being scanned that high
* watermark would be met at 100% efficiency.
*/
- if (kswapd_shrink_zone(zone, &sc, lru_pages,
- &nr_attempted))
+ if (kswapd_shrink_zone(zone, &sc,
+ lru_pages, shrinking_slab,
+ &nr_attempted))
raise_priority = false;
}
@@ -2943,6 +2952,9 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
pfmemalloc_watermark_ok(pgdat))
wake_up(&pgdat->pfmemalloc_wait);
+ /* Only shrink slab once per priority */
+ shrinking_slab = false;
+
/*
* Fragmentation may mean that the system cannot be rebalanced
* for high-order allocations in all zones. If twice the
@@ -2969,8 +2981,10 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
* Raise priority if scanning rate is too low or there was no
* progress in reclaiming pages
*/
- if (raise_priority || !sc.nr_reclaimed)
+ if (raise_priority || !sc.nr_reclaimed) {
sc.priority--;
+ shrinking_slab = true;
+ }
} while (sc.priority >= 1 &&
!pgdat_balanced(pgdat, order, *classzone_idx));
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 09/10] mm: vmscan: Check if kswapd should writepage once per pgdat scan
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Currently kswapd checks if it should start writepage as it shrinks
each zone without taking into consideration if the zone is balanced or
not. This is not wrong as such but it does not make much sense either.
This patch checks once per pgdat scan if kswapd should be writing pages.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Rik van Riel <riel@redhat.com>
---
mm/vmscan.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0fa588d..d45f6e2 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2846,6 +2846,13 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
}
/*
+ * If we're getting trouble reclaiming, start doing writepage
+ * even in laptop mode.
+ */
+ if (sc.priority < DEF_PRIORITY - 2)
+ sc.may_writepage = 1;
+
+ /*
* Now scan the zone in the dma->highmem direction, stopping
* at the last zone which needs scanning.
*
@@ -2917,13 +2924,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
raise_priority = false;
}
- /*
- * If we're getting trouble reclaiming, start doing
- * writepage even in laptop mode.
- */
- if (sc.priority < DEF_PRIORITY - 2)
- sc.may_writepage = 1;
-
if (zone->all_unreclaimable) {
if (end_zone && end_zone == i)
end_zone--;
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 09/10] mm: vmscan: Check if kswapd should writepage once per pgdat scan
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
Currently kswapd checks if it should start writepage as it shrinks
each zone without taking into consideration if the zone is balanced or
not. This is not wrong as such but it does not make much sense either.
This patch checks once per pgdat scan if kswapd should be writing pages.
Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Rik van Riel <riel@redhat.com>
---
mm/vmscan.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0fa588d..d45f6e2 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2846,6 +2846,13 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
}
/*
+ * If we're getting trouble reclaiming, start doing writepage
+ * even in laptop mode.
+ */
+ if (sc.priority < DEF_PRIORITY - 2)
+ sc.may_writepage = 1;
+
+ /*
* Now scan the zone in the dma->highmem direction, stopping
* at the last zone which needs scanning.
*
@@ -2917,13 +2924,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
raise_priority = false;
}
- /*
- * If we're getting trouble reclaiming, start doing
- * writepage even in laptop mode.
- */
- if (sc.priority < DEF_PRIORITY - 2)
- sc.may_writepage = 1;
-
if (zone->all_unreclaimable) {
if (end_zone && end_zone == i)
end_zone--;
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-11 19:57 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
balance_pgdat() is very long and some of the logic can and should
be internal to kswapd_shrink_zone(). Move it so the flow of
balance_pgdat() is marginally easier to follow.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 112 +++++++++++++++++++++++++++++-------------------------------
1 file changed, 55 insertions(+), 57 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index d45f6e2..2f1adf6 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2694,19 +2694,54 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* This is used to determine if the scanning priority needs to be raised.
*/
static bool kswapd_shrink_zone(struct zone *zone,
+ int classzone_idx,
struct scan_control *sc,
unsigned long lru_pages,
bool shrinking_slab,
unsigned long *nr_attempted)
{
+ int testorder = sc->order;
unsigned long nr_slab = 0;
+ unsigned long balance_gap;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
};
+ bool lowmem_pressure;
/* Reclaim above the high watermark. */
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+
+ /*
+ * Kswapd reclaims only single pages with compaction enabled. Trying
+ * too hard to reclaim until contiguous free pages have become
+ * available can hurt performance by evicting too much useful data
+ * from memory. Do not reclaim more than needed for compaction.
+ */
+ if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
+ compaction_suitable(zone, sc->order) !=
+ COMPACT_SKIPPED)
+ testorder = 0;
+
+ /*
+ * We put equal pressure on every zone, unless one zone has way too
+ * many pages free already. The "too many pages" is defined as the
+ * high wmark plus a "gap" where the gap is either the low
+ * watermark or 1% of the zone, whichever is smaller.
+ */
+ balance_gap = min(low_wmark_pages(zone),
+ (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
+ KSWAPD_ZONE_BALANCE_GAP_RATIO);
+
+ /*
+ * If there is no low memory pressure or the zone is balanced then no
+ * reclaim is necessary
+ */
+ lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
+ if (!lowmem_pressure && zone_balanced(zone, testorder,
+ balance_gap, classzone_idx))
+ return true;
+
shrink_zone(zone, sc);
/*
@@ -2727,6 +2762,18 @@ static bool kswapd_shrink_zone(struct zone *zone,
zone_clear_flag(zone, ZONE_WRITEBACK);
+ /*
+ * If a zone reaches its high watermark, consider it to be no longer
+ * congested. It's possible there are dirty pages backed by congested
+ * BDIs but as pressure is relieved, speculatively avoid congestion
+ * waits.
+ */
+ if (!zone->all_unreclaimable &&
+ zone_balanced(zone, testorder, 0, classzone_idx)) {
+ zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ }
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
@@ -2863,8 +2910,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int testorder;
- unsigned long balance_gap;
if (!populated_zone(zone))
continue;
@@ -2885,62 +2930,15 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
sc.nr_reclaimed += nr_soft_reclaimed;
/*
- * We put equal pressure on every zone, unless
- * one zone has way too many pages free
- * already. The "too many pages" is defined
- * as the high wmark plus a "gap" where the
- * gap is either the low watermark or 1%
- * of the zone, whichever is smaller.
- */
- balance_gap = min(low_wmark_pages(zone),
- (zone->managed_pages +
- KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
- KSWAPD_ZONE_BALANCE_GAP_RATIO);
- /*
- * Kswapd reclaims only single pages with compaction
- * enabled. Trying too hard to reclaim until contiguous
- * free pages have become available can hurt performance
- * by evicting too much useful data from memory.
- * Do not reclaim more than needed for compaction.
+ * There should be no need to raise the scanning
+ * priority if enough pages are already being scanned
+ * that that high watermark would be met at 100%
+ * efficiency.
*/
- testorder = order;
- if (IS_ENABLED(CONFIG_COMPACTION) && order &&
- compaction_suitable(zone, order) !=
- COMPACT_SKIPPED)
- testorder = 0;
-
- if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
- !zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- /*
- * There should be no need to raise the
- * scanning priority if enough pages are
- * already being scanned that high
- * watermark would be met at 100% efficiency.
- */
- if (kswapd_shrink_zone(zone, &sc,
- lru_pages, shrinking_slab,
- &nr_attempted))
- raise_priority = false;
- }
-
- if (zone->all_unreclaimable) {
- if (end_zone && end_zone == i)
- end_zone--;
- continue;
- }
-
- if (zone_balanced(zone, testorder, 0, end_zone))
- /*
- * If a zone reaches its high watermark,
- * consider it to be no longer congested. It's
- * possible there are dirty pages backed by
- * congested BDIs but as pressure is relieved,
- * speculatively avoid congestion waits
- * or writing pages from kswapd context.
- */
- zone_clear_flag(zone, ZONE_CONGESTED);
- zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ if (kswapd_shrink_zone(zone, end_zone, &sc,
+ lru_pages, shrinking_slab,
+ &nr_attempted))
+ raise_priority = false;
}
/*
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-04-11 19:57 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-11 19:57 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML, Mel Gorman
balance_pgdat() is very long and some of the logic can and should
be internal to kswapd_shrink_zone(). Move it so the flow of
balance_pgdat() is marginally easier to follow.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 112 +++++++++++++++++++++++++++++-------------------------------
1 file changed, 55 insertions(+), 57 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index d45f6e2..2f1adf6 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2694,19 +2694,54 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* This is used to determine if the scanning priority needs to be raised.
*/
static bool kswapd_shrink_zone(struct zone *zone,
+ int classzone_idx,
struct scan_control *sc,
unsigned long lru_pages,
bool shrinking_slab,
unsigned long *nr_attempted)
{
+ int testorder = sc->order;
unsigned long nr_slab = 0;
+ unsigned long balance_gap;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
};
+ bool lowmem_pressure;
/* Reclaim above the high watermark. */
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+
+ /*
+ * Kswapd reclaims only single pages with compaction enabled. Trying
+ * too hard to reclaim until contiguous free pages have become
+ * available can hurt performance by evicting too much useful data
+ * from memory. Do not reclaim more than needed for compaction.
+ */
+ if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
+ compaction_suitable(zone, sc->order) !=
+ COMPACT_SKIPPED)
+ testorder = 0;
+
+ /*
+ * We put equal pressure on every zone, unless one zone has way too
+ * many pages free already. The "too many pages" is defined as the
+ * high wmark plus a "gap" where the gap is either the low
+ * watermark or 1% of the zone, whichever is smaller.
+ */
+ balance_gap = min(low_wmark_pages(zone),
+ (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
+ KSWAPD_ZONE_BALANCE_GAP_RATIO);
+
+ /*
+ * If there is no low memory pressure or the zone is balanced then no
+ * reclaim is necessary
+ */
+ lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
+ if (!lowmem_pressure && zone_balanced(zone, testorder,
+ balance_gap, classzone_idx))
+ return true;
+
shrink_zone(zone, sc);
/*
@@ -2727,6 +2762,18 @@ static bool kswapd_shrink_zone(struct zone *zone,
zone_clear_flag(zone, ZONE_WRITEBACK);
+ /*
+ * If a zone reaches its high watermark, consider it to be no longer
+ * congested. It's possible there are dirty pages backed by congested
+ * BDIs but as pressure is relieved, speculatively avoid congestion
+ * waits.
+ */
+ if (!zone->all_unreclaimable &&
+ zone_balanced(zone, testorder, 0, classzone_idx)) {
+ zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ }
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
@@ -2863,8 +2910,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int testorder;
- unsigned long balance_gap;
if (!populated_zone(zone))
continue;
@@ -2885,62 +2930,15 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
sc.nr_reclaimed += nr_soft_reclaimed;
/*
- * We put equal pressure on every zone, unless
- * one zone has way too many pages free
- * already. The "too many pages" is defined
- * as the high wmark plus a "gap" where the
- * gap is either the low watermark or 1%
- * of the zone, whichever is smaller.
- */
- balance_gap = min(low_wmark_pages(zone),
- (zone->managed_pages +
- KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
- KSWAPD_ZONE_BALANCE_GAP_RATIO);
- /*
- * Kswapd reclaims only single pages with compaction
- * enabled. Trying too hard to reclaim until contiguous
- * free pages have become available can hurt performance
- * by evicting too much useful data from memory.
- * Do not reclaim more than needed for compaction.
+ * There should be no need to raise the scanning
+ * priority if enough pages are already being scanned
+ * that that high watermark would be met at 100%
+ * efficiency.
*/
- testorder = order;
- if (IS_ENABLED(CONFIG_COMPACTION) && order &&
- compaction_suitable(zone, order) !=
- COMPACT_SKIPPED)
- testorder = 0;
-
- if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
- !zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- /*
- * There should be no need to raise the
- * scanning priority if enough pages are
- * already being scanned that high
- * watermark would be met at 100% efficiency.
- */
- if (kswapd_shrink_zone(zone, &sc,
- lru_pages, shrinking_slab,
- &nr_attempted))
- raise_priority = false;
- }
-
- if (zone->all_unreclaimable) {
- if (end_zone && end_zone == i)
- end_zone--;
- continue;
- }
-
- if (zone_balanced(zone, testorder, 0, end_zone))
- /*
- * If a zone reaches its high watermark,
- * consider it to be no longer congested. It's
- * possible there are dirty pages backed by
- * congested BDIs but as pressure is relieved,
- * speculatively avoid congestion waits
- * or writing pages from kswapd context.
- */
- zone_clear_flag(zone, ZONE_CONGESTED);
- zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ if (kswapd_shrink_zone(zone, end_zone, &sc,
+ lru_pages, shrinking_slab,
+ &nr_attempted))
+ raise_priority = false;
}
/*
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* Re: [PATCH 02/10] mm: vmscan: Obey proportional scanning requirements for kswapd
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 15:01 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:01 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:50PM +0100, Mel Gorman wrote:
> @@ -1841,17 +1848,58 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
> lruvec, sc);
> }
> }
> +
> + if (nr_reclaimed < nr_to_reclaim || scan_adjusted)
> + continue;
> +
> /*
> - * On large memory systems, scan >> priority can become
> - * really large. This is fine for the starting priority;
> - * we want to put equal scanning pressure on each zone.
> - * However, if the VM has a harder time of freeing pages,
> - * with multiple processes reclaiming pages, the total
> - * freeing target can get unreasonably large.
> + * For global direct reclaim, reclaim only the number of pages
> + * requested. Less care is taken to scan proportionally as it
> + * is more important to minimise direct reclaim stall latency
> + * than it is to properly age the LRU lists.
> */
> - if (nr_reclaimed >= nr_to_reclaim &&
> - sc->priority < DEF_PRIORITY)
> + if (global_reclaim(sc) && !current_is_kswapd())
> break;
> +
> + /*
> + * For kswapd and memcg, reclaim at least the number of pages
> + * requested. Ensure that the anon and file LRUs shrink
> + * proportionally what was requested by get_scan_count(). We
> + * stop reclaiming one LRU and reduce the amount scanning
> + * proportional to the original scan target.
> + */
> + nr_file = nr[LRU_INACTIVE_FILE] + nr[LRU_ACTIVE_FILE];
> + nr_anon = nr[LRU_INACTIVE_ANON] + nr[LRU_ACTIVE_ANON];
> +
> + if (nr_file > nr_anon) {
> + unsigned long scan_target = targets[LRU_INACTIVE_ANON] +
> + targets[LRU_ACTIVE_ANON] + 1;
> + lru = LRU_BASE;
> + percentage = nr_anon * 100 / scan_target;
> + } else {
> + unsigned long scan_target = targets[LRU_INACTIVE_FILE] +
> + targets[LRU_ACTIVE_FILE] + 1;
> + lru = LRU_FILE;
> + percentage = nr_file * 100 / scan_target;
> + }
> +
> + /* Stop scanning the smaller of the LRU */
> + nr[lru] = 0;
> + nr[lru + LRU_ACTIVE] = 0;
> +
> + /*
> + * Recalculate the other LRU scan count based on its original
> + * scan target and the percentage scanning already complete
> + */
> + lru = (lru == LRU_FILE) ? LRU_BASE : LRU_FILE;
> + nr[lru] = targets[lru] * (100 - percentage) / 100;
> + nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
This doesn't seem right. Say percentage is 60, then
nr[lru] = targets[lru] * (100 - percentage) / 100;
sets nr[lru] to 40% of targets[lru], and so in
nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
targets[lru] - nr[lru] is 60% of targets[lru], making it bigger than
nr[lru], which is in turn subtracted from itself, i.e. it leaves the
remaining type at 0 if >= 50% of the other type were scanned, and at
half of the inverted scan percentage if less than 50% were scanned.
Would this be more sensible?
already_scanned = targets[lru] - nr[lru];
nr[lru] = targets[lru] * percentage / 100; /* adjusted original target */
nr[lru] -= min(nr[lru], already_scanned); /* minus work already done */
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 02/10] mm: vmscan: Obey proportional scanning requirements for kswapd
@ 2013-04-18 15:01 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:01 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:50PM +0100, Mel Gorman wrote:
> @@ -1841,17 +1848,58 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
> lruvec, sc);
> }
> }
> +
> + if (nr_reclaimed < nr_to_reclaim || scan_adjusted)
> + continue;
> +
> /*
> - * On large memory systems, scan >> priority can become
> - * really large. This is fine for the starting priority;
> - * we want to put equal scanning pressure on each zone.
> - * However, if the VM has a harder time of freeing pages,
> - * with multiple processes reclaiming pages, the total
> - * freeing target can get unreasonably large.
> + * For global direct reclaim, reclaim only the number of pages
> + * requested. Less care is taken to scan proportionally as it
> + * is more important to minimise direct reclaim stall latency
> + * than it is to properly age the LRU lists.
> */
> - if (nr_reclaimed >= nr_to_reclaim &&
> - sc->priority < DEF_PRIORITY)
> + if (global_reclaim(sc) && !current_is_kswapd())
> break;
> +
> + /*
> + * For kswapd and memcg, reclaim at least the number of pages
> + * requested. Ensure that the anon and file LRUs shrink
> + * proportionally what was requested by get_scan_count(). We
> + * stop reclaiming one LRU and reduce the amount scanning
> + * proportional to the original scan target.
> + */
> + nr_file = nr[LRU_INACTIVE_FILE] + nr[LRU_ACTIVE_FILE];
> + nr_anon = nr[LRU_INACTIVE_ANON] + nr[LRU_ACTIVE_ANON];
> +
> + if (nr_file > nr_anon) {
> + unsigned long scan_target = targets[LRU_INACTIVE_ANON] +
> + targets[LRU_ACTIVE_ANON] + 1;
> + lru = LRU_BASE;
> + percentage = nr_anon * 100 / scan_target;
> + } else {
> + unsigned long scan_target = targets[LRU_INACTIVE_FILE] +
> + targets[LRU_ACTIVE_FILE] + 1;
> + lru = LRU_FILE;
> + percentage = nr_file * 100 / scan_target;
> + }
> +
> + /* Stop scanning the smaller of the LRU */
> + nr[lru] = 0;
> + nr[lru + LRU_ACTIVE] = 0;
> +
> + /*
> + * Recalculate the other LRU scan count based on its original
> + * scan target and the percentage scanning already complete
> + */
> + lru = (lru == LRU_FILE) ? LRU_BASE : LRU_FILE;
> + nr[lru] = targets[lru] * (100 - percentage) / 100;
> + nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
This doesn't seem right. Say percentage is 60, then
nr[lru] = targets[lru] * (100 - percentage) / 100;
sets nr[lru] to 40% of targets[lru], and so in
nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
targets[lru] - nr[lru] is 60% of targets[lru], making it bigger than
nr[lru], which is in turn subtracted from itself, i.e. it leaves the
remaining type at 0 if >= 50% of the other type were scanned, and at
half of the inverted scan percentage if less than 50% were scanned.
Would this be more sensible?
already_scanned = targets[lru] - nr[lru];
nr[lru] = targets[lru] * percentage / 100; /* adjusted original target */
nr[lru] -= min(nr[lru], already_scanned); /* minus work already done */
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 03/10] mm: vmscan: Flatten kswapd priority loop
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 15:02 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:02 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:51PM +0100, Mel Gorman wrote:
> kswapd stops raising the scanning priority when at least SWAP_CLUSTER_MAX
> pages have been reclaimed or the pgdat is considered balanced. It then
> rechecks if it needs to restart at DEF_PRIORITY and whether high-order
> reclaim needs to be reset. This is not wrong per-se but it is confusing
> to follow and forcing kswapd to stay at DEF_PRIORITY may require several
> restarts before it has scanned enough pages to meet the high watermark even
> at 100% efficiency. This patch irons out the logic a bit by controlling
> when priority is raised and removing the "goto loop_again".
>
> This patch has kswapd raise the scanning priority until it is scanning
> enough pages that it could meet the high watermark in one shrink of the
> LRU lists if it is able to reclaim at 100% efficiency. It will not raise
> the scanning prioirty higher unless it is failing to reclaim any pages.
>
> To avoid infinite looping for high-order allocation requests kswapd will
> not reclaim for high-order allocations when it has reclaimed at least
> twice the number of pages as the allocation request.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 03/10] mm: vmscan: Flatten kswapd priority loop
@ 2013-04-18 15:02 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:02 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:51PM +0100, Mel Gorman wrote:
> kswapd stops raising the scanning priority when at least SWAP_CLUSTER_MAX
> pages have been reclaimed or the pgdat is considered balanced. It then
> rechecks if it needs to restart at DEF_PRIORITY and whether high-order
> reclaim needs to be reset. This is not wrong per-se but it is confusing
> to follow and forcing kswapd to stay at DEF_PRIORITY may require several
> restarts before it has scanned enough pages to meet the high watermark even
> at 100% efficiency. This patch irons out the logic a bit by controlling
> when priority is raised and removing the "goto loop_again".
>
> This patch has kswapd raise the scanning priority until it is scanning
> enough pages that it could meet the high watermark in one shrink of the
> LRU lists if it is able to reclaim at 100% efficiency. It will not raise
> the scanning prioirty higher unless it is failing to reclaim any pages.
>
> To avoid infinite looping for high-order allocation requests kswapd will
> not reclaim for high-order allocations when it has reclaimed at least
> twice the number of pages as the allocation request.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 04/10] mm: vmscan: Decide whether to compact the pgdat based on reclaim progress
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 15:09 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:09 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:52PM +0100, Mel Gorman wrote:
> @@ -2763,7 +2769,21 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> for (i = 0; i <= end_zone; i++) {
> struct zone *zone = pgdat->node_zones + i;
>
> + if (!populated_zone(zone))
> + continue;
> +
> lru_pages += zone_reclaimable_pages(zone);
> +
> + /*
> + * If any zone is currently balanced then kswapd will
> + * not call compaction as it is expected that the
> + * necessary pages are already available.
> + */
> + if (pgdat_needs_compaction &&
> + zone_watermark_ok(zone, order,
> + low_wmark_pages(zone),
> + *classzone_idx, 0))
> + pgdat_needs_compaction = false;
> }
>
> /*
> @@ -2832,7 +2852,8 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> * already being scanned that high
> * watermark would be met at 100% efficiency.
> */
> - if (kswapd_shrink_zone(zone, &sc, lru_pages))
> + if (kswapd_shrink_zone(zone, &sc, lru_pages,
> + &nr_attempted))
> raise_priority = false;
> }
There is the odd chance that the watermark is met after reclaim, would
it make sense to defer the pgdat_needs_compaction check? Not really a
big deal, though, so:
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 04/10] mm: vmscan: Decide whether to compact the pgdat based on reclaim progress
@ 2013-04-18 15:09 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:09 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:52PM +0100, Mel Gorman wrote:
> @@ -2763,7 +2769,21 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> for (i = 0; i <= end_zone; i++) {
> struct zone *zone = pgdat->node_zones + i;
>
> + if (!populated_zone(zone))
> + continue;
> +
> lru_pages += zone_reclaimable_pages(zone);
> +
> + /*
> + * If any zone is currently balanced then kswapd will
> + * not call compaction as it is expected that the
> + * necessary pages are already available.
> + */
> + if (pgdat_needs_compaction &&
> + zone_watermark_ok(zone, order,
> + low_wmark_pages(zone),
> + *classzone_idx, 0))
> + pgdat_needs_compaction = false;
> }
>
> /*
> @@ -2832,7 +2852,8 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> * already being scanned that high
> * watermark would be met at 100% efficiency.
> */
> - if (kswapd_shrink_zone(zone, &sc, lru_pages))
> + if (kswapd_shrink_zone(zone, &sc, lru_pages,
> + &nr_attempted))
> raise_priority = false;
> }
There is the odd chance that the watermark is met after reclaim, would
it make sense to defer the pgdat_needs_compaction check? Not really a
big deal, though, so:
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 05/10] mm: vmscan: Do not allow kswapd to scan at maximum priority
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 15:11 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:11 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:53PM +0100, Mel Gorman wrote:
> Page reclaim at priority 0 will scan the entire LRU as priority 0 is
> considered to be a near OOM condition. Kswapd can reach priority 0 quite
> easily if it is encountering a large number of pages it cannot reclaim
> such as pages under writeback. When this happens, kswapd reclaims very
> aggressively even though there may be no real risk of allocation failure
> or OOM.
>
> This patch prevents kswapd reaching priority 0 and trying to reclaim
> the world. Direct reclaimers will still reach priority 0 in the event
> of an OOM situation.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
> Acked-by: Rik van Riel <riel@redhat.com>
> Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 05/10] mm: vmscan: Do not allow kswapd to scan at maximum priority
@ 2013-04-18 15:11 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:11 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:53PM +0100, Mel Gorman wrote:
> Page reclaim at priority 0 will scan the entire LRU as priority 0 is
> considered to be a near OOM condition. Kswapd can reach priority 0 quite
> easily if it is encountering a large number of pages it cannot reclaim
> such as pages under writeback. When this happens, kswapd reclaims very
> aggressively even though there may be no real risk of allocation failure
> or OOM.
>
> This patch prevents kswapd reaching priority 0 and trying to reclaim
> the world. Direct reclaimers will still reach priority 0 in the event
> of an OOM situation.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
> Acked-by: Rik van Riel <riel@redhat.com>
> Reviewed-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 06/10] mm: vmscan: Have kswapd writeback pages based on dirty pages encountered, not priority
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 15:16 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:16 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:54PM +0100, Mel Gorman wrote:
> Currently kswapd queues dirty pages for writeback if scanning at an elevated
> priority but the priority kswapd scans at is not related to the number
> of unqueued dirty encountered. Since commit "mm: vmscan: Flatten kswapd
> priority loop", the priority is related to the size of the LRU and the
> zone watermark which is no indication as to whether kswapd should write
> pages or not.
>
> This patch tracks if an excessive number of unqueued dirty pages are being
> encountered at the end of the LRU. If so, it indicates that dirty pages
> are being recycled before flusher threads can clean them and flags the
> zone so that kswapd will start writing pages until the zone is balanced.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 06/10] mm: vmscan: Have kswapd writeback pages based on dirty pages encountered, not priority
@ 2013-04-18 15:16 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 15:16 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:54PM +0100, Mel Gorman wrote:
> Currently kswapd queues dirty pages for writeback if scanning at an elevated
> priority but the priority kswapd scans at is not related to the number
> of unqueued dirty encountered. Since commit "mm: vmscan: Flatten kswapd
> priority loop", the priority is related to the size of the LRU and the
> zone watermark which is no indication as to whether kswapd should write
> pages or not.
>
> This patch tracks if an excessive number of unqueued dirty pages are being
> encountered at the end of the LRU. If so, it indicates that dirty pages
> are being recycled before flusher threads can clean them and flags the
> zone so that kswapd will start writing pages until the zone is balanced.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 02/10] mm: vmscan: Obey proportional scanning requirements for kswapd
2013-04-18 15:01 ` Johannes Weiner
@ 2013-04-18 15:58 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-18 15:58 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 18, 2013 at 08:01:05AM -0700, Johannes Weiner wrote:
> On Thu, Apr 11, 2013 at 08:57:50PM +0100, Mel Gorman wrote:
> > @@ -1841,17 +1848,58 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
> > lruvec, sc);
> > }
> > }
> > +
> > + if (nr_reclaimed < nr_to_reclaim || scan_adjusted)
> > + continue;
> > +
> > /*
> > - * On large memory systems, scan >> priority can become
> > - * really large. This is fine for the starting priority;
> > - * we want to put equal scanning pressure on each zone.
> > - * However, if the VM has a harder time of freeing pages,
> > - * with multiple processes reclaiming pages, the total
> > - * freeing target can get unreasonably large.
> > + * For global direct reclaim, reclaim only the number of pages
> > + * requested. Less care is taken to scan proportionally as it
> > + * is more important to minimise direct reclaim stall latency
> > + * than it is to properly age the LRU lists.
> > */
> > - if (nr_reclaimed >= nr_to_reclaim &&
> > - sc->priority < DEF_PRIORITY)
> > + if (global_reclaim(sc) && !current_is_kswapd())
> > break;
> > +
> > + /*
> > + * For kswapd and memcg, reclaim at least the number of pages
> > + * requested. Ensure that the anon and file LRUs shrink
> > + * proportionally what was requested by get_scan_count(). We
> > + * stop reclaiming one LRU and reduce the amount scanning
> > + * proportional to the original scan target.
> > + */
> > + nr_file = nr[LRU_INACTIVE_FILE] + nr[LRU_ACTIVE_FILE];
> > + nr_anon = nr[LRU_INACTIVE_ANON] + nr[LRU_ACTIVE_ANON];
> > +
> > + if (nr_file > nr_anon) {
> > + unsigned long scan_target = targets[LRU_INACTIVE_ANON] +
> > + targets[LRU_ACTIVE_ANON] + 1;
> > + lru = LRU_BASE;
> > + percentage = nr_anon * 100 / scan_target;
> > + } else {
> > + unsigned long scan_target = targets[LRU_INACTIVE_FILE] +
> > + targets[LRU_ACTIVE_FILE] + 1;
> > + lru = LRU_FILE;
> > + percentage = nr_file * 100 / scan_target;
> > + }
> > +
> > + /* Stop scanning the smaller of the LRU */
> > + nr[lru] = 0;
> > + nr[lru + LRU_ACTIVE] = 0;
> > +
> > + /*
> > + * Recalculate the other LRU scan count based on its original
> > + * scan target and the percentage scanning already complete
> > + */
> > + lru = (lru == LRU_FILE) ? LRU_BASE : LRU_FILE;
> > + nr[lru] = targets[lru] * (100 - percentage) / 100;
> > + nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
>
> This doesn't seem right. Say percentage is 60, then
>
> nr[lru] = targets[lru] * (100 - percentage) / 100;
>
> sets nr[lru] to 40% of targets[lru], and so in
>
> nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
>
> targets[lru] - nr[lru] is 60% of targets[lru], making it bigger than
> nr[lru], which is in turn subtracted from itself, i.e. it leaves the
> remaining type at 0 if >= 50% of the other type were scanned, and at
> half of the inverted scan percentage if less than 50% were scanned.
>
> Would this be more sensible?
>
> already_scanned = targets[lru] - nr[lru];
> nr[lru] = targets[lru] * percentage / 100; /* adjusted original target */
> nr[lru] -= min(nr[lru], already_scanned); /* minus work already done */
Bah, yes, that was the intent as I was writing it. It's not what came
out my fingers. Thanks for the bashing with a clue stick.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 02/10] mm: vmscan: Obey proportional scanning requirements for kswapd
@ 2013-04-18 15:58 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-18 15:58 UTC (permalink / raw)
To: Johannes Weiner
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 18, 2013 at 08:01:05AM -0700, Johannes Weiner wrote:
> On Thu, Apr 11, 2013 at 08:57:50PM +0100, Mel Gorman wrote:
> > @@ -1841,17 +1848,58 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)
> > lruvec, sc);
> > }
> > }
> > +
> > + if (nr_reclaimed < nr_to_reclaim || scan_adjusted)
> > + continue;
> > +
> > /*
> > - * On large memory systems, scan >> priority can become
> > - * really large. This is fine for the starting priority;
> > - * we want to put equal scanning pressure on each zone.
> > - * However, if the VM has a harder time of freeing pages,
> > - * with multiple processes reclaiming pages, the total
> > - * freeing target can get unreasonably large.
> > + * For global direct reclaim, reclaim only the number of pages
> > + * requested. Less care is taken to scan proportionally as it
> > + * is more important to minimise direct reclaim stall latency
> > + * than it is to properly age the LRU lists.
> > */
> > - if (nr_reclaimed >= nr_to_reclaim &&
> > - sc->priority < DEF_PRIORITY)
> > + if (global_reclaim(sc) && !current_is_kswapd())
> > break;
> > +
> > + /*
> > + * For kswapd and memcg, reclaim at least the number of pages
> > + * requested. Ensure that the anon and file LRUs shrink
> > + * proportionally what was requested by get_scan_count(). We
> > + * stop reclaiming one LRU and reduce the amount scanning
> > + * proportional to the original scan target.
> > + */
> > + nr_file = nr[LRU_INACTIVE_FILE] + nr[LRU_ACTIVE_FILE];
> > + nr_anon = nr[LRU_INACTIVE_ANON] + nr[LRU_ACTIVE_ANON];
> > +
> > + if (nr_file > nr_anon) {
> > + unsigned long scan_target = targets[LRU_INACTIVE_ANON] +
> > + targets[LRU_ACTIVE_ANON] + 1;
> > + lru = LRU_BASE;
> > + percentage = nr_anon * 100 / scan_target;
> > + } else {
> > + unsigned long scan_target = targets[LRU_INACTIVE_FILE] +
> > + targets[LRU_ACTIVE_FILE] + 1;
> > + lru = LRU_FILE;
> > + percentage = nr_file * 100 / scan_target;
> > + }
> > +
> > + /* Stop scanning the smaller of the LRU */
> > + nr[lru] = 0;
> > + nr[lru + LRU_ACTIVE] = 0;
> > +
> > + /*
> > + * Recalculate the other LRU scan count based on its original
> > + * scan target and the percentage scanning already complete
> > + */
> > + lru = (lru == LRU_FILE) ? LRU_BASE : LRU_FILE;
> > + nr[lru] = targets[lru] * (100 - percentage) / 100;
> > + nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
>
> This doesn't seem right. Say percentage is 60, then
>
> nr[lru] = targets[lru] * (100 - percentage) / 100;
>
> sets nr[lru] to 40% of targets[lru], and so in
>
> nr[lru] -= min(nr[lru], (targets[lru] - nr[lru]));
>
> targets[lru] - nr[lru] is 60% of targets[lru], making it bigger than
> nr[lru], which is in turn subtracted from itself, i.e. it leaves the
> remaining type at 0 if >= 50% of the other type were scanned, and at
> half of the inverted scan percentage if less than 50% were scanned.
>
> Would this be more sensible?
>
> already_scanned = targets[lru] - nr[lru];
> nr[lru] = targets[lru] * percentage / 100; /* adjusted original target */
> nr[lru] -= min(nr[lru], already_scanned); /* minus work already done */
Bah, yes, that was the intent as I was writing it. It's not what came
out my fingers. Thanks for the bashing with a clue stick.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 08/10] mm: vmscan: Have kswapd shrink slab only once per priority
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 16:43 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 16:43 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:56PM +0100, Mel Gorman wrote:
> If kswaps fails to make progress but continues to shrink slab then it'll
> either discard all of slab or consume CPU uselessly scanning shrinkers.
> This patch causes kswapd to only call the shrinkers once per priority.
But the priority level changes _only_ when kswapd is not making
progress, so I don't see how this fixes this case.
On the other hand, what about shrinkable memory like dentries and
inodes that build up during a streaming IO load like a backup program?
Kswapd may be cooperating with the page allocator and never change
priority as it reclaims the continuous file page stream, but it won't
do the same for the stream of slab memory.
So if anything, I would expect us to lay off slab memory when lru
reclaim is struggling, but receive continuous aging and pushback as
long as lru reclaim is comfortably running alongside the workload.
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 08/10] mm: vmscan: Have kswapd shrink slab only once per priority
@ 2013-04-18 16:43 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 16:43 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:56PM +0100, Mel Gorman wrote:
> If kswaps fails to make progress but continues to shrink slab then it'll
> either discard all of slab or consume CPU uselessly scanning shrinkers.
> This patch causes kswapd to only call the shrinkers once per priority.
But the priority level changes _only_ when kswapd is not making
progress, so I don't see how this fixes this case.
On the other hand, what about shrinkable memory like dentries and
inodes that build up during a streaming IO load like a backup program?
Kswapd may be cooperating with the page allocator and never change
priority as it reclaims the continuous file page stream, but it won't
do the same for the stream of slab memory.
So if anything, I would expect us to lay off slab memory when lru
reclaim is struggling, but receive continuous aging and pushback as
long as lru reclaim is comfortably running alongside the workload.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 09/10] mm: vmscan: Check if kswapd should writepage once per pgdat scan
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 16:44 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 16:44 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:57PM +0100, Mel Gorman wrote:
> Currently kswapd checks if it should start writepage as it shrinks
> each zone without taking into consideration if the zone is balanced or
> not. This is not wrong as such but it does not make much sense either.
> This patch checks once per pgdat scan if kswapd should be writing pages.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
> Reviewed-by: Michal Hocko <mhocko@suse.cz>
> Acked-by: Rik van Riel <riel@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 09/10] mm: vmscan: Check if kswapd should writepage once per pgdat scan
@ 2013-04-18 16:44 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 16:44 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:57PM +0100, Mel Gorman wrote:
> Currently kswapd checks if it should start writepage as it shrinks
> each zone without taking into consideration if the zone is balanced or
> not. This is not wrong as such but it does not make much sense either.
> This patch checks once per pgdat scan if kswapd should be writing pages.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
> Reviewed-by: Michal Hocko <mhocko@suse.cz>
> Acked-by: Rik van Riel <riel@redhat.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-04-11 19:57 ` Mel Gorman
@ 2013-04-18 16:56 ` Johannes Weiner
-1 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 16:56 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:58PM +0100, Mel Gorman wrote:
> balance_pgdat() is very long and some of the logic can and should
> be internal to kswapd_shrink_zone(). Move it so the flow of
> balance_pgdat() is marginally easier to follow.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-04-18 16:56 ` Johannes Weiner
0 siblings, 0 replies; 54+ messages in thread
From: Johannes Weiner @ 2013-04-18 16:56 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, dormando, Michal Hocko, Kamezawa Hiroyuki,
Linux-MM, LKML
On Thu, Apr 11, 2013 at 08:57:58PM +0100, Mel Gorman wrote:
> balance_pgdat() is very long and some of the logic can and should
> be internal to kswapd_shrink_zone(). Move it so the flow of
> balance_pgdat() is marginally easier to follow.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-04-09 11:07 ` Mel Gorman
@ 2013-04-12 2:56 ` Rik van Riel
-1 siblings, 0 replies; 54+ messages in thread
From: Rik van Riel @ 2013-04-12 2:56 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Zlatko Calusic,
Johannes Weiner, dormando, Satoru Moriya, Michal Hocko, Linux-MM,
LKML
On 04/09/2013 07:07 AM, Mel Gorman wrote:
> balance_pgdat() is very long and some of the logic can and should
> be internal to kswapd_shrink_zone(). Move it so the flow of
> balance_pgdat() is marginally easier to follow.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
--
All rights reversed
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-04-12 2:56 ` Rik van Riel
0 siblings, 0 replies; 54+ messages in thread
From: Rik van Riel @ 2013-04-12 2:56 UTC (permalink / raw)
To: Mel Gorman
Cc: Andrew Morton, Jiri Slaby, Valdis Kletnieks, Zlatko Calusic,
Johannes Weiner, dormando, Satoru Moriya, Michal Hocko, Linux-MM,
LKML
On 04/09/2013 07:07 AM, Mel Gorman wrote:
> balance_pgdat() is very long and some of the logic can and should
> be internal to kswapd_shrink_zone(). Move it so the flow of
> balance_pgdat() is marginally easier to follow.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
--
All rights reversed
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-04-09 11:06 [PATCH 0/10] Reduce system disruption due to kswapd V2 Mel Gorman
@ 2013-04-09 11:07 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-09 11:07 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Satoru Moriya, Michal Hocko, Linux-MM,
LKML, Mel Gorman
balance_pgdat() is very long and some of the logic can and should
be internal to kswapd_shrink_zone(). Move it so the flow of
balance_pgdat() is marginally easier to follow.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 112 +++++++++++++++++++++++++++++-------------------------------
1 file changed, 55 insertions(+), 57 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6cd6435..00024d8 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2684,19 +2684,54 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* This is used to determine if the scanning priority needs to be raised.
*/
static bool kswapd_shrink_zone(struct zone *zone,
+ int classzone_idx,
struct scan_control *sc,
unsigned long lru_pages,
bool shrinking_slab,
unsigned long *nr_attempted)
{
+ int testorder = sc->order;
unsigned long nr_slab = 0;
+ unsigned long balance_gap;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
};
+ bool lowmem_pressure;
/* Reclaim above the high watermark. */
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+
+ /*
+ * Kswapd reclaims only single pages with compaction enabled. Trying
+ * too hard to reclaim until contiguous free pages have become
+ * available can hurt performance by evicting too much useful data
+ * from memory. Do not reclaim more than needed for compaction.
+ */
+ if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
+ compaction_suitable(zone, sc->order) !=
+ COMPACT_SKIPPED)
+ testorder = 0;
+
+ /*
+ * We put equal pressure on every zone, unless one zone has way too
+ * many pages free already. The "too many pages" is defined as the
+ * high wmark plus a "gap" where the gap is either the low
+ * watermark or 1% of the zone, whichever is smaller.
+ */
+ balance_gap = min(low_wmark_pages(zone),
+ (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
+ KSWAPD_ZONE_BALANCE_GAP_RATIO);
+
+ /*
+ * If there is no low memory pressure or the zone is balanced then no
+ * reclaim is necessary
+ */
+ lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
+ if (!lowmem_pressure && zone_balanced(zone, testorder,
+ balance_gap, classzone_idx))
+ return true;
+
shrink_zone(zone, sc);
/*
@@ -2717,6 +2752,18 @@ static bool kswapd_shrink_zone(struct zone *zone,
zone_clear_flag(zone, ZONE_WRITEBACK);
+ /*
+ * If a zone reaches its high watermark, consider it to be no longer
+ * congested. It's possible there are dirty pages backed by congested
+ * BDIs but as pressure is relieved, speculatively avoid congestion
+ * waits.
+ */
+ if (!zone->all_unreclaimable &&
+ zone_balanced(zone, testorder, 0, classzone_idx)) {
+ zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ }
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
@@ -2853,8 +2900,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int testorder;
- unsigned long balance_gap;
if (!populated_zone(zone))
continue;
@@ -2875,62 +2920,15 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
sc.nr_reclaimed += nr_soft_reclaimed;
/*
- * We put equal pressure on every zone, unless
- * one zone has way too many pages free
- * already. The "too many pages" is defined
- * as the high wmark plus a "gap" where the
- * gap is either the low watermark or 1%
- * of the zone, whichever is smaller.
- */
- balance_gap = min(low_wmark_pages(zone),
- (zone->managed_pages +
- KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
- KSWAPD_ZONE_BALANCE_GAP_RATIO);
- /*
- * Kswapd reclaims only single pages with compaction
- * enabled. Trying too hard to reclaim until contiguous
- * free pages have become available can hurt performance
- * by evicting too much useful data from memory.
- * Do not reclaim more than needed for compaction.
+ * There should be no need to raise the scanning
+ * priority if enough pages are already being scanned
+ * that that high watermark would be met at 100%
+ * efficiency.
*/
- testorder = order;
- if (IS_ENABLED(CONFIG_COMPACTION) && order &&
- compaction_suitable(zone, order) !=
- COMPACT_SKIPPED)
- testorder = 0;
-
- if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
- !zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- /*
- * There should be no need to raise the
- * scanning priority if enough pages are
- * already being scanned that high
- * watermark would be met at 100% efficiency.
- */
- if (kswapd_shrink_zone(zone, &sc,
- lru_pages, shrinking_slab,
- &nr_attempted))
- raise_priority = false;
- }
-
- if (zone->all_unreclaimable) {
- if (end_zone && end_zone == i)
- end_zone--;
- continue;
- }
-
- if (zone_balanced(zone, testorder, 0, end_zone))
- /*
- * If a zone reaches its high watermark,
- * consider it to be no longer congested. It's
- * possible there are dirty pages backed by
- * congested BDIs but as pressure is relieved,
- * speculatively avoid congestion waits
- * or writing pages from kswapd context.
- */
- zone_clear_flag(zone, ZONE_CONGESTED);
- zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ if (kswapd_shrink_zone(zone, end_zone, &sc,
+ lru_pages, shrinking_slab,
+ &nr_attempted))
+ raise_priority = false;
}
/*
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-04-09 11:07 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-04-09 11:07 UTC (permalink / raw)
To: Andrew Morton
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Satoru Moriya, Michal Hocko, Linux-MM,
LKML, Mel Gorman
balance_pgdat() is very long and some of the logic can and should
be internal to kswapd_shrink_zone(). Move it so the flow of
balance_pgdat() is marginally easier to follow.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 112 +++++++++++++++++++++++++++++-------------------------------
1 file changed, 55 insertions(+), 57 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 6cd6435..00024d8 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2684,19 +2684,54 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* This is used to determine if the scanning priority needs to be raised.
*/
static bool kswapd_shrink_zone(struct zone *zone,
+ int classzone_idx,
struct scan_control *sc,
unsigned long lru_pages,
bool shrinking_slab,
unsigned long *nr_attempted)
{
+ int testorder = sc->order;
unsigned long nr_slab = 0;
+ unsigned long balance_gap;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
};
+ bool lowmem_pressure;
/* Reclaim above the high watermark. */
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+
+ /*
+ * Kswapd reclaims only single pages with compaction enabled. Trying
+ * too hard to reclaim until contiguous free pages have become
+ * available can hurt performance by evicting too much useful data
+ * from memory. Do not reclaim more than needed for compaction.
+ */
+ if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
+ compaction_suitable(zone, sc->order) !=
+ COMPACT_SKIPPED)
+ testorder = 0;
+
+ /*
+ * We put equal pressure on every zone, unless one zone has way too
+ * many pages free already. The "too many pages" is defined as the
+ * high wmark plus a "gap" where the gap is either the low
+ * watermark or 1% of the zone, whichever is smaller.
+ */
+ balance_gap = min(low_wmark_pages(zone),
+ (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
+ KSWAPD_ZONE_BALANCE_GAP_RATIO);
+
+ /*
+ * If there is no low memory pressure or the zone is balanced then no
+ * reclaim is necessary
+ */
+ lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
+ if (!lowmem_pressure && zone_balanced(zone, testorder,
+ balance_gap, classzone_idx))
+ return true;
+
shrink_zone(zone, sc);
/*
@@ -2717,6 +2752,18 @@ static bool kswapd_shrink_zone(struct zone *zone,
zone_clear_flag(zone, ZONE_WRITEBACK);
+ /*
+ * If a zone reaches its high watermark, consider it to be no longer
+ * congested. It's possible there are dirty pages backed by congested
+ * BDIs but as pressure is relieved, speculatively avoid congestion
+ * waits.
+ */
+ if (!zone->all_unreclaimable &&
+ zone_balanced(zone, testorder, 0, classzone_idx)) {
+ zone_clear_flag(zone, ZONE_CONGESTED);
+ zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ }
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
@@ -2853,8 +2900,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int testorder;
- unsigned long balance_gap;
if (!populated_zone(zone))
continue;
@@ -2875,62 +2920,15 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
sc.nr_reclaimed += nr_soft_reclaimed;
/*
- * We put equal pressure on every zone, unless
- * one zone has way too many pages free
- * already. The "too many pages" is defined
- * as the high wmark plus a "gap" where the
- * gap is either the low watermark or 1%
- * of the zone, whichever is smaller.
- */
- balance_gap = min(low_wmark_pages(zone),
- (zone->managed_pages +
- KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
- KSWAPD_ZONE_BALANCE_GAP_RATIO);
- /*
- * Kswapd reclaims only single pages with compaction
- * enabled. Trying too hard to reclaim until contiguous
- * free pages have become available can hurt performance
- * by evicting too much useful data from memory.
- * Do not reclaim more than needed for compaction.
+ * There should be no need to raise the scanning
+ * priority if enough pages are already being scanned
+ * that that high watermark would be met at 100%
+ * efficiency.
*/
- testorder = order;
- if (IS_ENABLED(CONFIG_COMPACTION) && order &&
- compaction_suitable(zone, order) !=
- COMPACT_SKIPPED)
- testorder = 0;
-
- if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
- !zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- /*
- * There should be no need to raise the
- * scanning priority if enough pages are
- * already being scanned that high
- * watermark would be met at 100% efficiency.
- */
- if (kswapd_shrink_zone(zone, &sc,
- lru_pages, shrinking_slab,
- &nr_attempted))
- raise_priority = false;
- }
-
- if (zone->all_unreclaimable) {
- if (end_zone && end_zone == i)
- end_zone--;
- continue;
- }
-
- if (zone_balanced(zone, testorder, 0, end_zone))
- /*
- * If a zone reaches its high watermark,
- * consider it to be no longer congested. It's
- * possible there are dirty pages backed by
- * congested BDIs but as pressure is relieved,
- * speculatively avoid congestion waits
- * or writing pages from kswapd context.
- */
- zone_clear_flag(zone, ZONE_CONGESTED);
- zone_clear_flag(zone, ZONE_TAIL_LRU_DIRTY);
+ if (kswapd_shrink_zone(zone, end_zone, &sc,
+ lru_pages, shrinking_slab,
+ &nr_attempted))
+ raise_priority = false;
}
/*
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-03-21 17:18 ` Michal Hocko
@ 2013-03-21 18:13 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-03-21 18:13 UTC (permalink / raw)
To: Michal Hocko
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya, LKML
On Thu, Mar 21, 2013 at 06:18:04PM +0100, Michal Hocko wrote:
> On Sun 17-03-13 13:04:16, Mel Gorman wrote:
> > +
> > + /*
> > + * Kswapd reclaims only single pages with compaction enabled. Trying
> > + * too hard to reclaim until contiguous free pages have become
> > + * available can hurt performance by evicting too much useful data
> > + * from memory. Do not reclaim more than needed for compaction.
> > + */
> > + if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
> > + compaction_suitable(zone, sc->order) !=
> > + COMPACT_SKIPPED)
> > + testorder = 0;
> > +
> > + /*
> > + * We put equal pressure on every zone, unless one zone has way too
> > + * many pages free already. The "too many pages" is defined as the
> > + * high wmark plus a "gap" where the gap is either the low
> > + * watermark or 1% of the zone, whichever is smaller.
> > + */
> > + balance_gap = min(low_wmark_pages(zone),
> > + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> > + KSWAPD_ZONE_BALANCE_GAP_RATIO);
> > +
> > + /*
> > + * If there is no low memory pressure or the zone is balanced then no
> > + * reclaim is necessary
> > + */
> > + lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
> > + if (!(lowmem_pressure || !zone_balanced(zone, testorder,
> > + balance_gap, classzone_idx)))
>
> if (!lowmem_pressure && zone_balanced) would be less cryptic I guess
>
It would.
> > + return true;
> > +
> > shrink_zone(zone, sc);
> >
> > /*
> > @@ -2689,6 +2724,16 @@ static bool kswapd_shrink_zone(struct zone *zone,
> >
> > zone_clear_flag(zone, ZONE_WRITEBACK);
> >
> > + /*
> > + * If a zone reaches its high watermark, consider it to be no longer
> > + * congested. It's possible there are dirty pages backed by congested
> > + * BDIs but as pressure is relieved, speculatively avoid congestion
> > + * waits.
> > + */
> > + if (!zone->all_unreclaimable &&
> > + zone_balanced(zone, testorder, 0, classzone_idx))
> > + zone_clear_flag(zone, ZONE_CONGESTED);
> > +
> > return sc->nr_scanned >= sc->nr_to_reclaim;
> > }
> >
> > @@ -2821,8 +2866,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> > */
> > for (i = 0; i <= end_zone; i++) {
> > struct zone *zone = pgdat->node_zones + i;
> > - int testorder;
> > - unsigned long balance_gap;
> >
> > if (!populated_zone(zone))
> > continue;
> > @@ -2843,61 +2886,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> > sc.nr_reclaimed += nr_soft_reclaimed;
> >
> > /*
> > - * We put equal pressure on every zone, unless
> > - * one zone has way too many pages free
> > - * already. The "too many pages" is defined
> > - * as the high wmark plus a "gap" where the
> > - * gap is either the low watermark or 1%
> > - * of the zone, whichever is smaller.
> > - */
> > - balance_gap = min(low_wmark_pages(zone),
> > - (zone->managed_pages +
> > - KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> > - KSWAPD_ZONE_BALANCE_GAP_RATIO);
> > - /*
> > - * Kswapd reclaims only single pages with compaction
> > - * enabled. Trying too hard to reclaim until contiguous
> > - * free pages have become available can hurt performance
> > - * by evicting too much useful data from memory.
> > - * Do not reclaim more than needed for compaction.
> > + * There should be no need to raise the scanning
> > + * priority if enough pages are already being scanned
> > + * that that high watermark would be met at 100%
> > + * efficiency.
> > */
> > - testorder = order;
> > - if (IS_ENABLED(CONFIG_COMPACTION) && order &&
> > - compaction_suitable(zone, order) !=
> > - COMPACT_SKIPPED)
> > - testorder = 0;
> > -
> > - if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
> > - !zone_balanced(zone, testorder,
> > - balance_gap, end_zone)) {
> > - /*
> > - * There should be no need to raise the
> > - * scanning priority if enough pages are
> > - * already being scanned that that high
> > - * watermark would be met at 100% efficiency.
> > - */
> > - if (kswapd_shrink_zone(zone, &sc,
> > + if (kswapd_shrink_zone(zone, end_zone, &sc,
> > lru_pages, shrinking_slab))
> > raise_priority = false;
> >
> > - nr_to_reclaim += sc.nr_to_reclaim;
> > - }
> > -
> > - if (zone->all_unreclaimable) {
> > - if (end_zone && end_zone == i)
> > - end_zone--;
> > - continue;
> > - }
> > -
> > - if (zone_balanced(zone, testorder, 0, end_zone))
> > - /*
> > - * If a zone reaches its high watermark,
> > - * consider it to be no longer congested. It's
> > - * possible there are dirty pages backed by
> > - * congested BDIs but as pressure is relieved,
> > - * speculatively avoid congestion waits
> > - */
> > - zone_clear_flag(zone, ZONE_CONGESTED);
> > + nr_to_reclaim += sc.nr_to_reclaim;
> > }
>
> nr_to_reclaim is updated if the zone is balanced an no reclaim is done
> which break compaction condition AFAICS.
>
True, it only makes sense to account for the pages it actually attempted
to reclaim. Thanks.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-03-21 18:13 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-03-21 18:13 UTC (permalink / raw)
To: Michal Hocko
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya, LKML
On Thu, Mar 21, 2013 at 06:18:04PM +0100, Michal Hocko wrote:
> On Sun 17-03-13 13:04:16, Mel Gorman wrote:
> > +
> > + /*
> > + * Kswapd reclaims only single pages with compaction enabled. Trying
> > + * too hard to reclaim until contiguous free pages have become
> > + * available can hurt performance by evicting too much useful data
> > + * from memory. Do not reclaim more than needed for compaction.
> > + */
> > + if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
> > + compaction_suitable(zone, sc->order) !=
> > + COMPACT_SKIPPED)
> > + testorder = 0;
> > +
> > + /*
> > + * We put equal pressure on every zone, unless one zone has way too
> > + * many pages free already. The "too many pages" is defined as the
> > + * high wmark plus a "gap" where the gap is either the low
> > + * watermark or 1% of the zone, whichever is smaller.
> > + */
> > + balance_gap = min(low_wmark_pages(zone),
> > + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> > + KSWAPD_ZONE_BALANCE_GAP_RATIO);
> > +
> > + /*
> > + * If there is no low memory pressure or the zone is balanced then no
> > + * reclaim is necessary
> > + */
> > + lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
> > + if (!(lowmem_pressure || !zone_balanced(zone, testorder,
> > + balance_gap, classzone_idx)))
>
> if (!lowmem_pressure && zone_balanced) would be less cryptic I guess
>
It would.
> > + return true;
> > +
> > shrink_zone(zone, sc);
> >
> > /*
> > @@ -2689,6 +2724,16 @@ static bool kswapd_shrink_zone(struct zone *zone,
> >
> > zone_clear_flag(zone, ZONE_WRITEBACK);
> >
> > + /*
> > + * If a zone reaches its high watermark, consider it to be no longer
> > + * congested. It's possible there are dirty pages backed by congested
> > + * BDIs but as pressure is relieved, speculatively avoid congestion
> > + * waits.
> > + */
> > + if (!zone->all_unreclaimable &&
> > + zone_balanced(zone, testorder, 0, classzone_idx))
> > + zone_clear_flag(zone, ZONE_CONGESTED);
> > +
> > return sc->nr_scanned >= sc->nr_to_reclaim;
> > }
> >
> > @@ -2821,8 +2866,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> > */
> > for (i = 0; i <= end_zone; i++) {
> > struct zone *zone = pgdat->node_zones + i;
> > - int testorder;
> > - unsigned long balance_gap;
> >
> > if (!populated_zone(zone))
> > continue;
> > @@ -2843,61 +2886,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> > sc.nr_reclaimed += nr_soft_reclaimed;
> >
> > /*
> > - * We put equal pressure on every zone, unless
> > - * one zone has way too many pages free
> > - * already. The "too many pages" is defined
> > - * as the high wmark plus a "gap" where the
> > - * gap is either the low watermark or 1%
> > - * of the zone, whichever is smaller.
> > - */
> > - balance_gap = min(low_wmark_pages(zone),
> > - (zone->managed_pages +
> > - KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> > - KSWAPD_ZONE_BALANCE_GAP_RATIO);
> > - /*
> > - * Kswapd reclaims only single pages with compaction
> > - * enabled. Trying too hard to reclaim until contiguous
> > - * free pages have become available can hurt performance
> > - * by evicting too much useful data from memory.
> > - * Do not reclaim more than needed for compaction.
> > + * There should be no need to raise the scanning
> > + * priority if enough pages are already being scanned
> > + * that that high watermark would be met at 100%
> > + * efficiency.
> > */
> > - testorder = order;
> > - if (IS_ENABLED(CONFIG_COMPACTION) && order &&
> > - compaction_suitable(zone, order) !=
> > - COMPACT_SKIPPED)
> > - testorder = 0;
> > -
> > - if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
> > - !zone_balanced(zone, testorder,
> > - balance_gap, end_zone)) {
> > - /*
> > - * There should be no need to raise the
> > - * scanning priority if enough pages are
> > - * already being scanned that that high
> > - * watermark would be met at 100% efficiency.
> > - */
> > - if (kswapd_shrink_zone(zone, &sc,
> > + if (kswapd_shrink_zone(zone, end_zone, &sc,
> > lru_pages, shrinking_slab))
> > raise_priority = false;
> >
> > - nr_to_reclaim += sc.nr_to_reclaim;
> > - }
> > -
> > - if (zone->all_unreclaimable) {
> > - if (end_zone && end_zone == i)
> > - end_zone--;
> > - continue;
> > - }
> > -
> > - if (zone_balanced(zone, testorder, 0, end_zone))
> > - /*
> > - * If a zone reaches its high watermark,
> > - * consider it to be no longer congested. It's
> > - * possible there are dirty pages backed by
> > - * congested BDIs but as pressure is relieved,
> > - * speculatively avoid congestion waits
> > - */
> > - zone_clear_flag(zone, ZONE_CONGESTED);
> > + nr_to_reclaim += sc.nr_to_reclaim;
> > }
>
> nr_to_reclaim is updated if the zone is balanced an no reclaim is done
> which break compaction condition AFAICS.
>
True, it only makes sense to account for the pages it actually attempted
to reclaim. Thanks.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-03-17 13:04 ` Mel Gorman
@ 2013-03-21 17:18 ` Michal Hocko
-1 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2013-03-21 17:18 UTC (permalink / raw)
To: Mel Gorman
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya, LKML
On Sun 17-03-13 13:04:16, Mel Gorman wrote:
> balance_pgdat() is very long and some of the logic can and should
> be internal to kswapd_shrink_zone(). Move it so the flow of
> balance_pgdat() is marginally easier to follow.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
> ---
> mm/vmscan.c | 104 +++++++++++++++++++++++++++++-------------------------------
> 1 file changed, 51 insertions(+), 53 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 8c66e5a..d7cf384 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2660,18 +2660,53 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
> * reclaim or if the lack of process was due to pages under writeback.
> */
> static bool kswapd_shrink_zone(struct zone *zone,
> + int classzone_idx,
> struct scan_control *sc,
> unsigned long lru_pages,
> bool shrinking_slab)
> {
> + int testorder = sc->order;
> unsigned long nr_slab = 0;
> + unsigned long balance_gap;
> struct reclaim_state *reclaim_state = current->reclaim_state;
> struct shrink_control shrink = {
> .gfp_mask = sc->gfp_mask,
> };
> + bool lowmem_pressure;
>
> /* Reclaim above the high watermark. */
> sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
> +
> + /*
> + * Kswapd reclaims only single pages with compaction enabled. Trying
> + * too hard to reclaim until contiguous free pages have become
> + * available can hurt performance by evicting too much useful data
> + * from memory. Do not reclaim more than needed for compaction.
> + */
> + if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
> + compaction_suitable(zone, sc->order) !=
> + COMPACT_SKIPPED)
> + testorder = 0;
> +
> + /*
> + * We put equal pressure on every zone, unless one zone has way too
> + * many pages free already. The "too many pages" is defined as the
> + * high wmark plus a "gap" where the gap is either the low
> + * watermark or 1% of the zone, whichever is smaller.
> + */
> + balance_gap = min(low_wmark_pages(zone),
> + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> + KSWAPD_ZONE_BALANCE_GAP_RATIO);
> +
> + /*
> + * If there is no low memory pressure or the zone is balanced then no
> + * reclaim is necessary
> + */
> + lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
> + if (!(lowmem_pressure || !zone_balanced(zone, testorder,
> + balance_gap, classzone_idx)))
if (!lowmem_pressure && zone_balanced) would be less cryptic I guess
> + return true;
> +
> shrink_zone(zone, sc);
>
> /*
> @@ -2689,6 +2724,16 @@ static bool kswapd_shrink_zone(struct zone *zone,
>
> zone_clear_flag(zone, ZONE_WRITEBACK);
>
> + /*
> + * If a zone reaches its high watermark, consider it to be no longer
> + * congested. It's possible there are dirty pages backed by congested
> + * BDIs but as pressure is relieved, speculatively avoid congestion
> + * waits.
> + */
> + if (!zone->all_unreclaimable &&
> + zone_balanced(zone, testorder, 0, classzone_idx))
> + zone_clear_flag(zone, ZONE_CONGESTED);
> +
> return sc->nr_scanned >= sc->nr_to_reclaim;
> }
>
> @@ -2821,8 +2866,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> */
> for (i = 0; i <= end_zone; i++) {
> struct zone *zone = pgdat->node_zones + i;
> - int testorder;
> - unsigned long balance_gap;
>
> if (!populated_zone(zone))
> continue;
> @@ -2843,61 +2886,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> sc.nr_reclaimed += nr_soft_reclaimed;
>
> /*
> - * We put equal pressure on every zone, unless
> - * one zone has way too many pages free
> - * already. The "too many pages" is defined
> - * as the high wmark plus a "gap" where the
> - * gap is either the low watermark or 1%
> - * of the zone, whichever is smaller.
> - */
> - balance_gap = min(low_wmark_pages(zone),
> - (zone->managed_pages +
> - KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> - KSWAPD_ZONE_BALANCE_GAP_RATIO);
> - /*
> - * Kswapd reclaims only single pages with compaction
> - * enabled. Trying too hard to reclaim until contiguous
> - * free pages have become available can hurt performance
> - * by evicting too much useful data from memory.
> - * Do not reclaim more than needed for compaction.
> + * There should be no need to raise the scanning
> + * priority if enough pages are already being scanned
> + * that that high watermark would be met at 100%
> + * efficiency.
> */
> - testorder = order;
> - if (IS_ENABLED(CONFIG_COMPACTION) && order &&
> - compaction_suitable(zone, order) !=
> - COMPACT_SKIPPED)
> - testorder = 0;
> -
> - if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
> - !zone_balanced(zone, testorder,
> - balance_gap, end_zone)) {
> - /*
> - * There should be no need to raise the
> - * scanning priority if enough pages are
> - * already being scanned that that high
> - * watermark would be met at 100% efficiency.
> - */
> - if (kswapd_shrink_zone(zone, &sc,
> + if (kswapd_shrink_zone(zone, end_zone, &sc,
> lru_pages, shrinking_slab))
> raise_priority = false;
>
> - nr_to_reclaim += sc.nr_to_reclaim;
> - }
> -
> - if (zone->all_unreclaimable) {
> - if (end_zone && end_zone == i)
> - end_zone--;
> - continue;
> - }
> -
> - if (zone_balanced(zone, testorder, 0, end_zone))
> - /*
> - * If a zone reaches its high watermark,
> - * consider it to be no longer congested. It's
> - * possible there are dirty pages backed by
> - * congested BDIs but as pressure is relieved,
> - * speculatively avoid congestion waits
> - */
> - zone_clear_flag(zone, ZONE_CONGESTED);
> + nr_to_reclaim += sc.nr_to_reclaim;
> }
nr_to_reclaim is updated if the zone is balanced an no reclaim is done
which break compaction condition AFAICS.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-03-21 17:18 ` Michal Hocko
0 siblings, 0 replies; 54+ messages in thread
From: Michal Hocko @ 2013-03-21 17:18 UTC (permalink / raw)
To: Mel Gorman
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya, LKML
On Sun 17-03-13 13:04:16, Mel Gorman wrote:
> balance_pgdat() is very long and some of the logic can and should
> be internal to kswapd_shrink_zone(). Move it so the flow of
> balance_pgdat() is marginally easier to follow.
>
> Signed-off-by: Mel Gorman <mgorman@suse.de>
> ---
> mm/vmscan.c | 104 +++++++++++++++++++++++++++++-------------------------------
> 1 file changed, 51 insertions(+), 53 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 8c66e5a..d7cf384 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2660,18 +2660,53 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
> * reclaim or if the lack of process was due to pages under writeback.
> */
> static bool kswapd_shrink_zone(struct zone *zone,
> + int classzone_idx,
> struct scan_control *sc,
> unsigned long lru_pages,
> bool shrinking_slab)
> {
> + int testorder = sc->order;
> unsigned long nr_slab = 0;
> + unsigned long balance_gap;
> struct reclaim_state *reclaim_state = current->reclaim_state;
> struct shrink_control shrink = {
> .gfp_mask = sc->gfp_mask,
> };
> + bool lowmem_pressure;
>
> /* Reclaim above the high watermark. */
> sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
> +
> + /*
> + * Kswapd reclaims only single pages with compaction enabled. Trying
> + * too hard to reclaim until contiguous free pages have become
> + * available can hurt performance by evicting too much useful data
> + * from memory. Do not reclaim more than needed for compaction.
> + */
> + if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
> + compaction_suitable(zone, sc->order) !=
> + COMPACT_SKIPPED)
> + testorder = 0;
> +
> + /*
> + * We put equal pressure on every zone, unless one zone has way too
> + * many pages free already. The "too many pages" is defined as the
> + * high wmark plus a "gap" where the gap is either the low
> + * watermark or 1% of the zone, whichever is smaller.
> + */
> + balance_gap = min(low_wmark_pages(zone),
> + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> + KSWAPD_ZONE_BALANCE_GAP_RATIO);
> +
> + /*
> + * If there is no low memory pressure or the zone is balanced then no
> + * reclaim is necessary
> + */
> + lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
> + if (!(lowmem_pressure || !zone_balanced(zone, testorder,
> + balance_gap, classzone_idx)))
if (!lowmem_pressure && zone_balanced) would be less cryptic I guess
> + return true;
> +
> shrink_zone(zone, sc);
>
> /*
> @@ -2689,6 +2724,16 @@ static bool kswapd_shrink_zone(struct zone *zone,
>
> zone_clear_flag(zone, ZONE_WRITEBACK);
>
> + /*
> + * If a zone reaches its high watermark, consider it to be no longer
> + * congested. It's possible there are dirty pages backed by congested
> + * BDIs but as pressure is relieved, speculatively avoid congestion
> + * waits.
> + */
> + if (!zone->all_unreclaimable &&
> + zone_balanced(zone, testorder, 0, classzone_idx))
> + zone_clear_flag(zone, ZONE_CONGESTED);
> +
> return sc->nr_scanned >= sc->nr_to_reclaim;
> }
>
> @@ -2821,8 +2866,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> */
> for (i = 0; i <= end_zone; i++) {
> struct zone *zone = pgdat->node_zones + i;
> - int testorder;
> - unsigned long balance_gap;
>
> if (!populated_zone(zone))
> continue;
> @@ -2843,61 +2886,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
> sc.nr_reclaimed += nr_soft_reclaimed;
>
> /*
> - * We put equal pressure on every zone, unless
> - * one zone has way too many pages free
> - * already. The "too many pages" is defined
> - * as the high wmark plus a "gap" where the
> - * gap is either the low watermark or 1%
> - * of the zone, whichever is smaller.
> - */
> - balance_gap = min(low_wmark_pages(zone),
> - (zone->managed_pages +
> - KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> - KSWAPD_ZONE_BALANCE_GAP_RATIO);
> - /*
> - * Kswapd reclaims only single pages with compaction
> - * enabled. Trying too hard to reclaim until contiguous
> - * free pages have become available can hurt performance
> - * by evicting too much useful data from memory.
> - * Do not reclaim more than needed for compaction.
> + * There should be no need to raise the scanning
> + * priority if enough pages are already being scanned
> + * that that high watermark would be met at 100%
> + * efficiency.
> */
> - testorder = order;
> - if (IS_ENABLED(CONFIG_COMPACTION) && order &&
> - compaction_suitable(zone, order) !=
> - COMPACT_SKIPPED)
> - testorder = 0;
> -
> - if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
> - !zone_balanced(zone, testorder,
> - balance_gap, end_zone)) {
> - /*
> - * There should be no need to raise the
> - * scanning priority if enough pages are
> - * already being scanned that that high
> - * watermark would be met at 100% efficiency.
> - */
> - if (kswapd_shrink_zone(zone, &sc,
> + if (kswapd_shrink_zone(zone, end_zone, &sc,
> lru_pages, shrinking_slab))
> raise_priority = false;
>
> - nr_to_reclaim += sc.nr_to_reclaim;
> - }
> -
> - if (zone->all_unreclaimable) {
> - if (end_zone && end_zone == i)
> - end_zone--;
> - continue;
> - }
> -
> - if (zone_balanced(zone, testorder, 0, end_zone))
> - /*
> - * If a zone reaches its high watermark,
> - * consider it to be no longer congested. It's
> - * possible there are dirty pages backed by
> - * congested BDIs but as pressure is relieved,
> - * speculatively avoid congestion waits
> - */
> - zone_clear_flag(zone, ZONE_CONGESTED);
> + nr_to_reclaim += sc.nr_to_reclaim;
> }
nr_to_reclaim is updated if the zone is balanced an no reclaim is done
which break compaction condition AFAICS.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-03-17 14:55 ` Andi Kleen
@ 2013-03-17 15:25 ` Mel Gorman
-1 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-03-17 15:25 UTC (permalink / raw)
To: Andi Kleen
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya,
Michal Hocko, LKML
On Sun, Mar 17, 2013 at 07:55:54AM -0700, Andi Kleen wrote:
> Mel Gorman <mgorman@suse.de> writes:
>
> > +
> > + /*
> > + * We put equal pressure on every zone, unless one zone has way too
> > + * many pages free already. The "too many pages" is defined as the
> > + * high wmark plus a "gap" where the gap is either the low
> > + * watermark or 1% of the zone, whichever is smaller.
> > + */
> > + balance_gap = min(low_wmark_pages(zone),
> > + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> > + KSWAPD_ZONE_BALANCE_GAP_RATIO);
>
> Don't like those hard coded tunables. 1% of a 512GB node can be still
> quite a lot. Shouldn't the low watermark be enough?
>
1% of 512G would be lot but in that case, it'll use the low watermark as
the balance gap.
--
Mel Gorman
SUSE Labs
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-03-17 15:25 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-03-17 15:25 UTC (permalink / raw)
To: Andi Kleen
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya,
Michal Hocko, LKML
On Sun, Mar 17, 2013 at 07:55:54AM -0700, Andi Kleen wrote:
> Mel Gorman <mgorman@suse.de> writes:
>
> > +
> > + /*
> > + * We put equal pressure on every zone, unless one zone has way too
> > + * many pages free already. The "too many pages" is defined as the
> > + * high wmark plus a "gap" where the gap is either the low
> > + * watermark or 1% of the zone, whichever is smaller.
> > + */
> > + balance_gap = min(low_wmark_pages(zone),
> > + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> > + KSWAPD_ZONE_BALANCE_GAP_RATIO);
>
> Don't like those hard coded tunables. 1% of a 512GB node can be still
> quite a lot. Shouldn't the low watermark be enough?
>
1% of 512G would be lot but in that case, it'll use the low watermark as
the balance gap.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-03-17 13:04 ` Mel Gorman
@ 2013-03-17 14:55 ` Andi Kleen
-1 siblings, 0 replies; 54+ messages in thread
From: Andi Kleen @ 2013-03-17 14:55 UTC (permalink / raw)
To: Mel Gorman
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya,
Michal Hocko, LKML
Mel Gorman <mgorman@suse.de> writes:
> +
> + /*
> + * We put equal pressure on every zone, unless one zone has way too
> + * many pages free already. The "too many pages" is defined as the
> + * high wmark plus a "gap" where the gap is either the low
> + * watermark or 1% of the zone, whichever is smaller.
> + */
> + balance_gap = min(low_wmark_pages(zone),
> + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> + KSWAPD_ZONE_BALANCE_GAP_RATIO);
Don't like those hard coded tunables. 1% of a 512GB node can be still
quite a lot. Shouldn't the low watermark be enough?
-Andi
--
ak@linux.intel.com -- Speaking for myself only
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-03-17 14:55 ` Andi Kleen
0 siblings, 0 replies; 54+ messages in thread
From: Andi Kleen @ 2013-03-17 14:55 UTC (permalink / raw)
To: Mel Gorman
Cc: Linux-MM, Jiri Slaby, Valdis Kletnieks, Rik van Riel,
Zlatko Calusic, Johannes Weiner, dormando, Satoru Moriya,
Michal Hocko, LKML
Mel Gorman <mgorman@suse.de> writes:
> +
> + /*
> + * We put equal pressure on every zone, unless one zone has way too
> + * many pages free already. The "too many pages" is defined as the
> + * high wmark plus a "gap" where the gap is either the low
> + * watermark or 1% of the zone, whichever is smaller.
> + */
> + balance_gap = min(low_wmark_pages(zone),
> + (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
> + KSWAPD_ZONE_BALANCE_GAP_RATIO);
Don't like those hard coded tunables. 1% of a 512GB node can be still
quite a lot. Shouldn't the low watermark be enough?
-Andi
--
ak@linux.intel.com -- Speaking for myself only
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
2013-03-17 13:04 [RFC PATCH 0/8] Reduce system disruption due to kswapd Mel Gorman
@ 2013-03-17 13:04 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-03-17 13:04 UTC (permalink / raw)
To: Linux-MM
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Satoru Moriya, Michal Hocko, LKML,
Mel Gorman
balance_pgdat() is very long and some of the logic can and should
be internal to kswapd_shrink_zone(). Move it so the flow of
balance_pgdat() is marginally easier to follow.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 104 +++++++++++++++++++++++++++++-------------------------------
1 file changed, 51 insertions(+), 53 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 8c66e5a..d7cf384 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2660,18 +2660,53 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* reclaim or if the lack of process was due to pages under writeback.
*/
static bool kswapd_shrink_zone(struct zone *zone,
+ int classzone_idx,
struct scan_control *sc,
unsigned long lru_pages,
bool shrinking_slab)
{
+ int testorder = sc->order;
unsigned long nr_slab = 0;
+ unsigned long balance_gap;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
};
+ bool lowmem_pressure;
/* Reclaim above the high watermark. */
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+
+ /*
+ * Kswapd reclaims only single pages with compaction enabled. Trying
+ * too hard to reclaim until contiguous free pages have become
+ * available can hurt performance by evicting too much useful data
+ * from memory. Do not reclaim more than needed for compaction.
+ */
+ if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
+ compaction_suitable(zone, sc->order) !=
+ COMPACT_SKIPPED)
+ testorder = 0;
+
+ /*
+ * We put equal pressure on every zone, unless one zone has way too
+ * many pages free already. The "too many pages" is defined as the
+ * high wmark plus a "gap" where the gap is either the low
+ * watermark or 1% of the zone, whichever is smaller.
+ */
+ balance_gap = min(low_wmark_pages(zone),
+ (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
+ KSWAPD_ZONE_BALANCE_GAP_RATIO);
+
+ /*
+ * If there is no low memory pressure or the zone is balanced then no
+ * reclaim is necessary
+ */
+ lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
+ if (!(lowmem_pressure || !zone_balanced(zone, testorder,
+ balance_gap, classzone_idx)))
+ return true;
+
shrink_zone(zone, sc);
/*
@@ -2689,6 +2724,16 @@ static bool kswapd_shrink_zone(struct zone *zone,
zone_clear_flag(zone, ZONE_WRITEBACK);
+ /*
+ * If a zone reaches its high watermark, consider it to be no longer
+ * congested. It's possible there are dirty pages backed by congested
+ * BDIs but as pressure is relieved, speculatively avoid congestion
+ * waits.
+ */
+ if (!zone->all_unreclaimable &&
+ zone_balanced(zone, testorder, 0, classzone_idx))
+ zone_clear_flag(zone, ZONE_CONGESTED);
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
@@ -2821,8 +2866,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int testorder;
- unsigned long balance_gap;
if (!populated_zone(zone))
continue;
@@ -2843,61 +2886,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
sc.nr_reclaimed += nr_soft_reclaimed;
/*
- * We put equal pressure on every zone, unless
- * one zone has way too many pages free
- * already. The "too many pages" is defined
- * as the high wmark plus a "gap" where the
- * gap is either the low watermark or 1%
- * of the zone, whichever is smaller.
- */
- balance_gap = min(low_wmark_pages(zone),
- (zone->managed_pages +
- KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
- KSWAPD_ZONE_BALANCE_GAP_RATIO);
- /*
- * Kswapd reclaims only single pages with compaction
- * enabled. Trying too hard to reclaim until contiguous
- * free pages have become available can hurt performance
- * by evicting too much useful data from memory.
- * Do not reclaim more than needed for compaction.
+ * There should be no need to raise the scanning
+ * priority if enough pages are already being scanned
+ * that that high watermark would be met at 100%
+ * efficiency.
*/
- testorder = order;
- if (IS_ENABLED(CONFIG_COMPACTION) && order &&
- compaction_suitable(zone, order) !=
- COMPACT_SKIPPED)
- testorder = 0;
-
- if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
- !zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- /*
- * There should be no need to raise the
- * scanning priority if enough pages are
- * already being scanned that that high
- * watermark would be met at 100% efficiency.
- */
- if (kswapd_shrink_zone(zone, &sc,
+ if (kswapd_shrink_zone(zone, end_zone, &sc,
lru_pages, shrinking_slab))
raise_priority = false;
- nr_to_reclaim += sc.nr_to_reclaim;
- }
-
- if (zone->all_unreclaimable) {
- if (end_zone && end_zone == i)
- end_zone--;
- continue;
- }
-
- if (zone_balanced(zone, testorder, 0, end_zone))
- /*
- * If a zone reaches its high watermark,
- * consider it to be no longer congested. It's
- * possible there are dirty pages backed by
- * congested BDIs but as pressure is relieved,
- * speculatively avoid congestion waits
- */
- zone_clear_flag(zone, ZONE_CONGESTED);
+ nr_to_reclaim += sc.nr_to_reclaim;
}
/*
--
1.8.1.4
^ permalink raw reply related [flat|nested] 54+ messages in thread
* [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone()
@ 2013-03-17 13:04 ` Mel Gorman
0 siblings, 0 replies; 54+ messages in thread
From: Mel Gorman @ 2013-03-17 13:04 UTC (permalink / raw)
To: Linux-MM
Cc: Jiri Slaby, Valdis Kletnieks, Rik van Riel, Zlatko Calusic,
Johannes Weiner, dormando, Satoru Moriya, Michal Hocko, LKML,
Mel Gorman
balance_pgdat() is very long and some of the logic can and should
be internal to kswapd_shrink_zone(). Move it so the flow of
balance_pgdat() is marginally easier to follow.
Signed-off-by: Mel Gorman <mgorman@suse.de>
---
mm/vmscan.c | 104 +++++++++++++++++++++++++++++-------------------------------
1 file changed, 51 insertions(+), 53 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 8c66e5a..d7cf384 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2660,18 +2660,53 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
* reclaim or if the lack of process was due to pages under writeback.
*/
static bool kswapd_shrink_zone(struct zone *zone,
+ int classzone_idx,
struct scan_control *sc,
unsigned long lru_pages,
bool shrinking_slab)
{
+ int testorder = sc->order;
unsigned long nr_slab = 0;
+ unsigned long balance_gap;
struct reclaim_state *reclaim_state = current->reclaim_state;
struct shrink_control shrink = {
.gfp_mask = sc->gfp_mask,
};
+ bool lowmem_pressure;
/* Reclaim above the high watermark. */
sc->nr_to_reclaim = max(SWAP_CLUSTER_MAX, high_wmark_pages(zone));
+
+ /*
+ * Kswapd reclaims only single pages with compaction enabled. Trying
+ * too hard to reclaim until contiguous free pages have become
+ * available can hurt performance by evicting too much useful data
+ * from memory. Do not reclaim more than needed for compaction.
+ */
+ if (IS_ENABLED(CONFIG_COMPACTION) && sc->order &&
+ compaction_suitable(zone, sc->order) !=
+ COMPACT_SKIPPED)
+ testorder = 0;
+
+ /*
+ * We put equal pressure on every zone, unless one zone has way too
+ * many pages free already. The "too many pages" is defined as the
+ * high wmark plus a "gap" where the gap is either the low
+ * watermark or 1% of the zone, whichever is smaller.
+ */
+ balance_gap = min(low_wmark_pages(zone),
+ (zone->managed_pages + KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
+ KSWAPD_ZONE_BALANCE_GAP_RATIO);
+
+ /*
+ * If there is no low memory pressure or the zone is balanced then no
+ * reclaim is necessary
+ */
+ lowmem_pressure = (buffer_heads_over_limit && is_highmem(zone));
+ if (!(lowmem_pressure || !zone_balanced(zone, testorder,
+ balance_gap, classzone_idx)))
+ return true;
+
shrink_zone(zone, sc);
/*
@@ -2689,6 +2724,16 @@ static bool kswapd_shrink_zone(struct zone *zone,
zone_clear_flag(zone, ZONE_WRITEBACK);
+ /*
+ * If a zone reaches its high watermark, consider it to be no longer
+ * congested. It's possible there are dirty pages backed by congested
+ * BDIs but as pressure is relieved, speculatively avoid congestion
+ * waits.
+ */
+ if (!zone->all_unreclaimable &&
+ zone_balanced(zone, testorder, 0, classzone_idx))
+ zone_clear_flag(zone, ZONE_CONGESTED);
+
return sc->nr_scanned >= sc->nr_to_reclaim;
}
@@ -2821,8 +2866,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
- int testorder;
- unsigned long balance_gap;
if (!populated_zone(zone))
continue;
@@ -2843,61 +2886,16 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
sc.nr_reclaimed += nr_soft_reclaimed;
/*
- * We put equal pressure on every zone, unless
- * one zone has way too many pages free
- * already. The "too many pages" is defined
- * as the high wmark plus a "gap" where the
- * gap is either the low watermark or 1%
- * of the zone, whichever is smaller.
- */
- balance_gap = min(low_wmark_pages(zone),
- (zone->managed_pages +
- KSWAPD_ZONE_BALANCE_GAP_RATIO-1) /
- KSWAPD_ZONE_BALANCE_GAP_RATIO);
- /*
- * Kswapd reclaims only single pages with compaction
- * enabled. Trying too hard to reclaim until contiguous
- * free pages have become available can hurt performance
- * by evicting too much useful data from memory.
- * Do not reclaim more than needed for compaction.
+ * There should be no need to raise the scanning
+ * priority if enough pages are already being scanned
+ * that that high watermark would be met at 100%
+ * efficiency.
*/
- testorder = order;
- if (IS_ENABLED(CONFIG_COMPACTION) && order &&
- compaction_suitable(zone, order) !=
- COMPACT_SKIPPED)
- testorder = 0;
-
- if ((buffer_heads_over_limit && is_highmem_idx(i)) ||
- !zone_balanced(zone, testorder,
- balance_gap, end_zone)) {
- /*
- * There should be no need to raise the
- * scanning priority if enough pages are
- * already being scanned that that high
- * watermark would be met at 100% efficiency.
- */
- if (kswapd_shrink_zone(zone, &sc,
+ if (kswapd_shrink_zone(zone, end_zone, &sc,
lru_pages, shrinking_slab))
raise_priority = false;
- nr_to_reclaim += sc.nr_to_reclaim;
- }
-
- if (zone->all_unreclaimable) {
- if (end_zone && end_zone == i)
- end_zone--;
- continue;
- }
-
- if (zone_balanced(zone, testorder, 0, end_zone))
- /*
- * If a zone reaches its high watermark,
- * consider it to be no longer congested. It's
- * possible there are dirty pages backed by
- * congested BDIs but as pressure is relieved,
- * speculatively avoid congestion waits
- */
- zone_clear_flag(zone, ZONE_CONGESTED);
+ nr_to_reclaim += sc.nr_to_reclaim;
}
/*
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 54+ messages in thread
end of thread, other threads:[~2013-04-18 16:57 UTC | newest]
Thread overview: 54+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-04-11 19:57 [PATCH 0/10] Reduce system disruption due to kswapd V3 Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-11 19:57 ` [PATCH 01/10] mm: vmscan: Limit the number of pages kswapd reclaims at each priority Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-11 19:57 ` [PATCH 02/10] mm: vmscan: Obey proportional scanning requirements for kswapd Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 15:01 ` Johannes Weiner
2013-04-18 15:01 ` Johannes Weiner
2013-04-18 15:58 ` Mel Gorman
2013-04-18 15:58 ` Mel Gorman
2013-04-11 19:57 ` [PATCH 03/10] mm: vmscan: Flatten kswapd priority loop Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 15:02 ` Johannes Weiner
2013-04-18 15:02 ` Johannes Weiner
2013-04-11 19:57 ` [PATCH 04/10] mm: vmscan: Decide whether to compact the pgdat based on reclaim progress Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 15:09 ` Johannes Weiner
2013-04-18 15:09 ` Johannes Weiner
2013-04-11 19:57 ` [PATCH 05/10] mm: vmscan: Do not allow kswapd to scan at maximum priority Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 15:11 ` Johannes Weiner
2013-04-18 15:11 ` Johannes Weiner
2013-04-11 19:57 ` [PATCH 06/10] mm: vmscan: Have kswapd writeback pages based on dirty pages encountered, not priority Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 15:16 ` Johannes Weiner
2013-04-18 15:16 ` Johannes Weiner
2013-04-11 19:57 ` [PATCH 07/10] mm: vmscan: Block kswapd if it is encountering pages under writeback Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-11 19:57 ` [PATCH 08/10] mm: vmscan: Have kswapd shrink slab only once per priority Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 16:43 ` Johannes Weiner
2013-04-18 16:43 ` Johannes Weiner
2013-04-11 19:57 ` [PATCH 09/10] mm: vmscan: Check if kswapd should writepage once per pgdat scan Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 16:44 ` Johannes Weiner
2013-04-18 16:44 ` Johannes Weiner
2013-04-11 19:57 ` [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone() Mel Gorman
2013-04-11 19:57 ` Mel Gorman
2013-04-18 16:56 ` Johannes Weiner
2013-04-18 16:56 ` Johannes Weiner
-- strict thread matches above, loose matches on Subject: below --
2013-04-09 11:06 [PATCH 0/10] Reduce system disruption due to kswapd V2 Mel Gorman
2013-04-09 11:07 ` [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone() Mel Gorman
2013-04-09 11:07 ` Mel Gorman
2013-04-12 2:56 ` Rik van Riel
2013-04-12 2:56 ` Rik van Riel
2013-03-17 13:04 [RFC PATCH 0/8] Reduce system disruption due to kswapd Mel Gorman
2013-03-17 13:04 ` [PATCH 10/10] mm: vmscan: Move logic from balance_pgdat() to kswapd_shrink_zone() Mel Gorman
2013-03-17 13:04 ` Mel Gorman
2013-03-17 14:55 ` Andi Kleen
2013-03-17 14:55 ` Andi Kleen
2013-03-17 15:25 ` Mel Gorman
2013-03-17 15:25 ` Mel Gorman
2013-03-21 17:18 ` Michal Hocko
2013-03-21 17:18 ` Michal Hocko
2013-03-21 18:13 ` Mel Gorman
2013-03-21 18:13 ` Mel Gorman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.