From: Bron Gondwana <brong@fastmail.fm>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>,
Bron Gondwana <brong@fastmail.fm>,
Christian Kujau <lists@nerdbynature.de>,
Andrew Morton <akpm@linux-foundation.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
robm@fastmail.fm
Subject: [PATCH 1/1] mm: add dirty_highmem option
Date: Thu, 22 Nov 2007 14:42:04 +1100 [thread overview]
Message-ID: <20071122034204.GA14079@brong.net> (raw)
In-Reply-To: <alpine.LFD.0.9999.0711151300250.4260@woody.linux-foundation.org>
On Thu, Nov 15, 2007 at 01:14:32PM -0800, Linus Torvalds wrote:
> Examples of non-broken solutions:
> (a) always use lowmem sizes (what we do now)
> (b) always use total mem sizes (sane but potentially dangerous: but the
> VM pressure should work! It has serious bounce-buffer issues, though,
> which is why I think it's crazy even if it's otherwise consistent)
>
> Btw, I actually suspect that while (a) is what we do now, for the specific
> case that Bron has, we could have a /proc/sys/vm option to just enable
> (b). So we don't have to have just one consistent model, we can allow odd
> users (and Bron sounds like one - sorry Bron ;) to just force other, odd,
> but consistent models.
A 32 bit machine with HIGHMEM64 enabled running DCC has an MMAPed file
of approximately 2Gb size which contains a hash format that is written
"randomly" by the dbclean process. On 2.6.16 this process took a few
minutes. With lowmem only accounting of dirty ratios, this takes about
12 hours of 100% disk IO, all random writes.
This patch includes some code cleanup from Linus and a toggle in
/proc/sys/vm/dirty_highmem which can be set to 1 to add the highmem
back to the total available memory count.
Signed-off-by: Bron Gondwana <brong@fastmail.fm>
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c
===================================================================
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/mm/page-writeback.c 2007-11-22 01:48:20.000000000 +0000
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c 2007-11-22 02:42:04.000000000 +0000
@@ -70,6 +70,12 @@ static inline long sync_writeback_pages(
int dirty_background_ratio = 5;
/*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_dirty_highmem is true
+ */
+int vm_dirty_highmem;
+
+/*
* The generator of dirty data starts writeback at this percentage
*/
int vm_dirty_ratio = 10;
@@ -153,7 +159,8 @@ static unsigned long determine_dirtyable
x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);
- x -= highmem_dirtyable_memory(x);
+ if (!vm_dirty_highmem)
+ x -= highmem_dirtyable_memory(x);
return x + 1; /* Ensure that we never return 0 */
}
@@ -163,20 +170,12 @@ get_dirty_limits(long *pbackground, long
{
int background_ratio; /* Percentages */
int dirty_ratio;
- int unmapped_ratio;
long background;
long dirty;
unsigned long available_memory = determine_dirtyable_memory();
struct task_struct *tsk;
- unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
- global_page_state(NR_ANON_PAGES)) * 100) /
- available_memory;
-
dirty_ratio = vm_dirty_ratio;
- if (dirty_ratio > unmapped_ratio / 2)
- dirty_ratio = unmapped_ratio / 2;
-
if (dirty_ratio < 5)
dirty_ratio = 5;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
===================================================================
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/include/linux/writeback.h 2007-10-09 20:31:38.000000000 +0000
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h 2007-11-22 01:48:21.000000000 +0000
@@ -92,6 +92,7 @@ void throttle_vm_writeout(gfp_t gfp_mask
/* These are exported to sysctl. */
extern int dirty_background_ratio;
+extern int vm_dirty_highmem;
extern int vm_dirty_ratio;
extern int dirty_writeback_interval;
extern int dirty_expire_interval;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c
===================================================================
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/kernel/sysctl.c 2007-10-09 20:31:38.000000000 +0000
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c 2007-11-22 01:48:21.000000000 +0000
@@ -776,6 +776,7 @@ static ctl_table kern_table[] = {
/* Constants for minimum and maximum testing in vm_table.
We use these as one-element integer vectors. */
static int zero;
+static int one = 1;
static int two = 2;
static int one_hundred = 100;
@@ -1066,6 +1067,19 @@ static ctl_table vm_table[] = {
.extra1 = &zero,
},
#endif
+#ifdef CONFIG_HIGHMEM
+ {
+ .ctl_name = CTL_UNNUMBERED,
+ .procname = "dirty_highmem",
+ .data = &vm_dirty_highmem,
+ .maxlen = sizeof(vm_dirty_highmem),
+ .mode = 0644,
+ .proc_handler = &proc_dointvec_minmax,
+ .strategy = &sysctl_intvec,
+ .extra1 = &zero,
+ .extra2 = &one,
+ },
+#endif
/*
* NOTE: do not add new entries to this table unless you have read
* Documentation/sysctl/ctl_unnumbered.txt
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/filesystems/proc.txt
===================================================================
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/Documentation/filesystems/proc.txt 2007-11-22 02:32:36.000000000 +0000
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/filesystems/proc.txt 2007-11-22 02:39:11.000000000 +0000
@@ -1229,6 +1229,18 @@ dirty_background_ratio
Contains, as a percentage of total system memory, the number of pages at which
the pdflush background writeback daemon will start writing out dirty data.
+dirty_highmem
+-------------
+
+Contains, as a boolean, a switch to allow highmem to be counted as
+part of the "available" memory against which the dirty ratios will be
+applied.
+
+Setting this to 1 can be useful on 32 bit machines where you want to make
+random changes within an MMAPed file that is larger than your available
+lowmem, however it is potentially dangerous and has serious bounce-buffer
+issues.
+
dirty_ratio
-----------------
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/sysctl/vm.txt
===================================================================
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/Documentation/sysctl/vm.txt 2007-11-22 02:31:32.000000000 +0000
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/sysctl/vm.txt 2007-11-22 02:32:31.000000000 +0000
@@ -18,6 +18,7 @@ files can be found in mm/swap.c.
Currently, these files are in /proc/sys/vm:
- overcommit_memory
- page-cluster
+- dirty_highmem
- dirty_ratio
- dirty_background_ratio
- dirty_expire_centisecs
@@ -36,10 +37,10 @@ Currently, these files are in /proc/sys/
==============================================================
-dirty_ratio, dirty_background_ratio, dirty_expire_centisecs,
-dirty_writeback_centisecs, vfs_cache_pressure, laptop_mode,
-block_dump, swap_token_timeout, drop-caches,
-hugepages_treat_as_movable:
+dirty_highmem, dirty_ratio, dirty_background_ratio,
+dirty_expire_centisecs, dirty_writeback_centisecs,
+vfs_cache_pressure, laptop_mode, block_dump,
+swap_token_timeout, drop-caches, hugepages_treat_as_movable:
See Documentation/filesystems/proc.txt
next prev parent reply other threads:[~2007-11-22 3:42 UTC|newest]
Thread overview: 268+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-11-13 6:42 [BUG] New Kernel Bugs Natalie Protasevich
2007-11-13 11:15 ` Andrew Morton
2007-11-13 11:24 ` Jens Axboe
2007-11-13 11:33 ` Evgeniy Polyakov
2007-11-13 11:39 ` David Miller
2007-11-13 11:49 ` Andrew Morton
2007-11-13 11:58 ` David Miller
2007-11-13 12:12 ` Andrew Morton
2007-11-13 12:32 ` David Miller
2007-11-13 19:02 ` Andrew Morton
2007-11-13 20:00 ` Christian Kujau
2007-11-13 21:04 ` Andrew Morton
2007-11-13 16:56 ` Nick Piggin
2007-11-14 19:54 ` Linus Torvalds
2007-11-14 22:22 ` Heikki Orsila
2007-11-14 23:05 ` Linus Torvalds
2007-11-13 21:37 ` Adrian Bunk
2007-11-13 21:56 ` Christian Kujau
2007-11-15 4:07 ` Bron Gondwana
2007-11-15 4:24 ` Linus Torvalds
2007-11-15 5:25 ` Bron Gondwana
2007-11-15 5:35 ` Linus Torvalds
2007-11-15 5:53 ` Linus Torvalds
2007-11-15 11:50 ` mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs) Bron Gondwana
2007-11-15 16:32 ` Linus Torvalds
2007-11-15 19:40 ` Peter Zijlstra
2007-11-15 20:44 ` Peter Zijlstra
2007-11-15 20:56 ` Linus Torvalds
2007-11-15 20:59 ` Peter Zijlstra
2007-11-15 21:12 ` Peter Zijlstra
2007-11-15 21:14 ` Linus Torvalds
2007-11-15 21:26 ` Linus Torvalds
2007-11-15 21:26 ` Peter Zijlstra
2007-11-15 21:47 ` Linus Torvalds
2007-11-15 22:11 ` Chris Friesen
2007-11-15 22:31 ` Linus Torvalds
2007-11-15 22:24 ` Rob Mueller
2007-11-18 23:13 ` Daniel Phillips
2007-11-19 3:41 ` Bron Gondwana
2007-11-16 0:48 ` Alan Cox
2007-11-21 21:25 ` Jan Engelhardt
2007-11-19 3:54 ` Bron Gondwana
2007-11-22 3:42 ` Bron Gondwana [this message]
2007-11-26 17:53 ` [PATCH 1/1] mm: add dirty_highmem option Linus Torvalds
2007-11-27 1:30 ` Bron Gondwana
2007-11-27 4:54 ` Andrew Morton
2007-11-27 5:24 ` Bron Gondwana
2007-11-27 5:53 ` Andrew Morton
2007-11-27 12:10 ` dirty highmem calculation sysctl name (Was: [PATCH 1/1] mm: add dirty_highmem option) Bron Gondwana
2007-11-27 13:06 ` [PATCH] mm/page-writeback - highmem_is_dirtyable option (replaces dirty_highmem patch) Bron Gondwana
2007-11-21 23:51 ` mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs) Bron Gondwana
2007-11-22 2:16 ` Bron Gondwana
2007-11-13 19:32 ` [BUG] New Kernel Bugs Russell King
2007-11-13 20:13 ` Adrian Bunk
2007-11-13 23:29 ` Russell King
2007-11-13 23:38 ` Andrew Morton
2007-11-13 20:52 ` Andrew Morton
2007-11-13 22:18 ` Russell King
2007-11-13 22:32 ` Andrew Morton
2007-11-13 23:09 ` Russell King
2007-11-13 23:17 ` Andrew Morton
2007-11-14 1:55 ` David Miller
2007-11-14 2:27 ` Andrew Morton
2007-11-14 3:47 ` David Miller
2007-11-14 8:30 ` Russell King
2007-11-14 9:55 ` Russell King
2007-11-14 10:07 ` David Miller
2007-11-14 11:46 ` [alsa-devel] " Rene Herman
2007-11-14 11:56 ` David Miller
2007-11-14 12:01 ` David Miller
2007-11-14 8:25 ` Moderated list (Was: Re: [BUG] New Kernel Bugs) Takashi Iwai
2007-11-14 12:21 ` Rene Herman
2007-11-14 9:47 ` Takashi Iwai
2007-11-14 23:23 ` Moderated list David Miller
2007-11-15 6:09 ` Rene Herman
2007-11-14 12:12 ` [alsa-devel] [BUG] New Kernel Bugs Rene Herman
2007-11-14 12:09 ` Rene Herman
2007-11-15 4:16 ` Bron Gondwana
2007-11-15 5:59 ` Rene Herman
2007-11-15 12:02 ` Bron Gondwana
2007-11-15 12:26 ` Rene Herman
2007-11-15 13:00 ` Jörn Engel
2007-11-15 14:29 ` Rene Herman
2007-11-15 13:17 ` Olivier Galibert
2007-11-15 9:34 ` Takashi Iwai
2007-11-14 19:44 ` Russell King
2007-11-16 22:16 ` Use *poof* for linux-omap (Was: [BUG] New Kernel Bugs) Tony Lindgren
2007-11-17 0:45 ` Use *poof* for linux-omap David Miller
2007-11-18 20:01 ` Tony Lindgren
2007-11-14 5:56 ` [BUG] New Kernel Bugs Sam Ravnborg
2007-11-14 5:59 ` Sam Ravnborg
2007-11-14 6:13 ` David Miller
2007-11-13 13:40 ` Ingo Molnar
2007-11-13 14:08 ` Mark Lord
2007-11-13 15:24 ` Giacomo A. Catenazzi
2007-11-13 15:57 ` Ray Lee
2007-11-13 17:01 ` Adrian Bunk
2007-11-13 17:50 ` Romano Giannetti
2007-11-13 22:03 ` Frans Pop
2007-11-13 15:52 ` Benoit Boissinot
2007-11-13 16:49 ` Ingo Molnar
2007-11-13 17:13 ` Theodore Tso
2007-11-13 17:30 ` Alan Cox
2007-11-13 17:33 ` Larry Finger
2007-11-13 18:55 ` Theodore Tso
2007-11-13 20:07 ` Larry Finger
2007-11-13 17:56 ` Adrian Bunk
2007-11-13 18:57 ` Gabriel C
2007-11-14 0:41 ` Denys Vlasenko
2007-11-14 0:39 ` Denys Vlasenko
2007-11-14 7:27 ` Adrian Bunk
2007-11-14 7:46 ` Denys Vlasenko
2007-11-14 13:30 ` Matthew Wilcox
2007-11-14 13:35 ` Hannes Reinecke
2007-11-14 21:39 ` Denys Vlasenko
2007-11-14 21:58 ` Gabriel C
2007-11-14 18:27 ` Kok, Auke
2007-11-14 16:55 ` Jan Evert van Grootheest
2007-11-14 23:23 ` Daniel Barkalow
2007-11-15 15:30 ` Theodore Tso
2007-11-15 16:19 ` Daniel Barkalow
2007-11-16 8:20 ` Romano Giannetti
2007-11-16 18:20 ` Daniel Barkalow
2007-11-16 19:46 ` Theodore Tso
2007-11-17 12:20 ` Adrian Bunk
2007-11-18 18:01 ` Theodore Tso
2007-11-13 16:46 ` Ingo Molnar
2007-11-13 17:50 ` Mark Lord
2007-11-13 18:12 ` Adrian Bunk
2007-11-13 18:18 ` Mark Lord
2007-11-13 18:36 ` Adrian Bunk
2007-11-13 18:47 ` Mark Lord
2007-11-13 19:04 ` Adrian Bunk
2007-11-13 19:12 ` Mark Lord
2007-11-13 19:30 ` Adrian Bunk
2007-11-13 19:46 ` Russell King
2007-11-13 20:04 ` Adrian Bunk
2007-11-13 19:26 ` Mark Lord
2007-11-13 20:00 ` Adrian Bunk
2007-11-13 20:13 ` Mark Lord
2007-11-13 21:20 ` Adrian Bunk
2007-11-13 21:12 ` Alan Cox
2007-11-14 0:52 ` Chuck Ebbert
2007-11-14 1:11 ` Stephen Hemminger
2007-11-14 2:10 ` Andrew Morton
2007-11-14 1:10 ` David Miller
2007-11-14 1:18 ` Peter Stuge
2007-11-13 18:17 ` Peter Zijlstra
2007-11-13 18:39 ` Matthew Wilcox
2007-11-13 18:43 ` Mark Lord
2007-11-13 18:49 ` Matthew Wilcox
2007-11-13 18:54 ` Mark Lord
2007-11-13 22:09 ` Rafael J. Wysocki
2007-11-14 14:30 ` Ingo Molnar
2007-11-14 14:49 ` Larry Finger
2007-11-18 12:44 ` size of git repository (was Re: [BUG] New Kernel Bugs) Pavel Machek
2007-11-18 12:58 ` Rene Herman
2007-11-18 14:35 ` James Bottomley
2007-11-18 15:19 ` Rene Herman
2007-11-18 14:56 ` Ingo Molnar
2007-11-19 4:43 ` Willy Tarreau
2007-11-13 19:37 ` [BUG] New Kernel Bugs Russell King
2007-11-13 20:18 ` Mark Lord
2007-11-13 21:33 ` Jörn Engel
2007-11-13 21:56 ` Andrew Morton
2007-11-13 22:24 ` Jörn Engel
2007-11-13 22:43 ` Andrew Morton
2007-11-13 22:29 ` Mark Lord
2007-11-13 23:40 ` Russell King
2007-11-14 1:56 ` David Miller
2007-11-14 0:34 ` Denys Vlasenko
2007-11-15 3:06 ` Neil Brown
2007-11-13 16:55 ` Randy Dunlap
2007-11-14 14:08 ` Ingo Molnar
2007-11-14 17:38 ` Randy Dunlap
2007-11-14 18:23 ` J. Bruce Fields
2007-11-15 2:50 ` Neil Brown
2007-11-16 0:05 ` J. Bruce Fields
2007-11-14 20:16 ` Ingo Molnar
2007-11-14 20:29 ` Randy Dunlap
2007-11-14 20:37 ` Ingo Molnar
2007-11-14 21:05 ` Randy Dunlap
2007-11-14 19:56 ` David Miller
2007-11-14 20:09 ` James Bottomley
2007-11-14 20:54 ` Ingo Molnar
2007-11-14 20:48 ` Ingo Molnar
2007-11-14 21:05 ` david
2007-11-13 11:47 ` Jarek Poplawski
2007-11-13 13:58 ` Mark Lord
2007-11-13 14:18 ` Mark Lord
2007-11-13 16:08 ` Thomas Gleixner
2007-11-13 16:07 ` Thomas Gleixner
2007-11-13 17:47 ` Mark Lord
2007-11-15 16:32 ` [BUG] Strange 1-second pauses during Resume-from-RAM Mark Lord
2007-11-15 16:49 ` Ray Lee
2007-11-15 16:51 ` Mark Lord
2007-11-15 16:53 ` Mark Lord
2007-11-15 18:14 ` Pavel Machek
2007-11-15 17:31 ` Mark Lord
2007-11-15 19:34 ` Ingo Molnar
2007-11-15 19:36 ` Ingo Molnar
2007-11-15 22:23 ` Mark Lord
2007-11-16 5:55 ` Ingo Molnar
2007-11-16 7:15 ` Ingo Molnar
2007-11-16 8:21 ` Ingo Molnar
2007-11-16 11:23 ` Ingo Molnar
2007-11-16 11:53 ` Mike Galbraith
2007-11-16 12:43 ` Ingo Molnar
2007-11-16 12:58 ` [patch] snd hda suspend latency: shorten codec read Ingo Molnar
2007-11-16 13:31 ` Rafael J. Wysocki
2007-11-16 14:21 ` Takashi Iwai
2007-11-16 19:06 ` [BUG] Strange 1-second pauses during Resume-from-RAM Mark Lord
2007-11-16 18:35 ` Mark Lord
2007-11-30 20:12 ` Mark Lord
2007-11-30 12:56 ` Jörn Engel
2007-11-30 13:35 ` Ingo Molnar
2007-11-30 13:43 ` Ingo Molnar
2007-11-30 18:35 ` Jörn Engel
2007-11-30 18:46 ` Ingo Molnar
2007-12-01 15:16 ` Jörn Engel
2007-12-01 18:32 ` Ingo Molnar
2007-12-01 20:47 ` Jörn Engel
2007-12-01 20:54 ` Ingo Molnar
2007-12-01 23:41 ` Jörn Engel
2007-12-02 8:56 ` Ingo Molnar
2007-12-02 11:31 ` Jörn Engel
2007-12-02 12:31 ` Jörn Engel
2007-12-02 13:57 ` Ingo Molnar
2007-12-02 14:46 ` Jörn Engel
2007-12-02 15:44 ` Ingo Molnar
2007-12-02 13:57 ` Ingo Molnar
2007-12-02 14:11 ` Jörn Engel
2007-12-02 15:47 ` Ingo Molnar
2007-12-02 19:55 ` Jörn Engel
2007-12-02 20:07 ` Ingo Molnar
2007-12-02 20:30 ` Jörn Engel
2007-12-02 20:45 ` Ingo Molnar
2007-12-02 21:08 ` Jörn Engel
2007-12-02 21:10 ` Jörn Engel
2007-12-02 21:19 ` Ingo Molnar
2007-12-03 0:57 ` Jörn Engel
2007-12-04 0:06 ` Jörn Engel
2007-12-04 9:34 ` Ingo Molnar
2007-11-30 15:49 ` Jörn Engel
2007-11-15 20:27 ` Rafael J. Wysocki
2007-11-18 16:10 ` Mark Lord
2007-11-18 16:21 ` Ingo Molnar
2007-11-18 17:37 ` Mark Lord
2007-11-13 17:54 ` [BUG] New Kernel Bugs Mark Lord
2007-11-13 22:46 ` Thomas Gleixner
2007-11-13 23:37 ` Mark Lord
2007-11-13 18:10 ` Russell King
2007-11-13 18:25 ` Alan Cox
2007-11-13 22:34 ` Russell King
2007-11-15 20:16 ` Ben Dooks
2007-11-13 15:21 ` Bartlomiej Zolnierkiewicz
2007-11-13 15:33 ` James Bottomley
2007-11-13 16:43 ` Randy Dunlap
2007-11-13 17:46 ` Martin Bligh
2007-11-13 18:47 ` Andrew Morton
2007-11-14 5:07 ` David Miller
2007-11-13 15:36 ` Alan Cox
2007-11-13 17:49 ` Jan Kara
2007-11-13 18:04 ` Russell King
2007-11-14 12:46 ` Jiri Kosina
2007-11-14 13:24 ` Pavel Machek
2007-11-14 14:14 ` Fabio Comolli
2007-11-14 19:52 ` Russell King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20071122034204.GA14079@brong.net \
--to=brong@fastmail.fm \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lists@nerdbynature.de \
--cc=robm@fastmail.fm \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).