Off-Topic Write cache disabling?

Off-Topic Write cache disabling?

[Weedy]

Is there a kernel option or sysfs toggle that disables write caching?
Or forces the kernel to commit everything constantly.

------
I don't really want to join another ML, especially a higher traffic
one just to ask this when it only bugs me sometimes. But I'll shut up
if this is unwanted.
------

I use a similar kernel config with respect to selected options on all
my systems but this only effect my laptop.
On any of my personal system or system I have remote access too
nr_dirty drifts up and down and nr_writeback stays around 0 (assuming
the system isn't working hard).
On my laptop both nr_dirty and nr_writeback stay at 0. I can make them
go up to 10ish if I untar something but then almost immediately go
back to 0. If I didn't know better I would swear dirty_*_centisecs or
something was set to a near instant commit interval but I haven't
found evidence of that. The hard drive light blinks almost constantly
once a second, even if I'm at a X login screen.
As I said this doesn't bug me most of the time but if I let my FF
session get too large or start multiple VMs anything that might make
me swap a little, the machine pretty much dies from IOWAIT. Which I'm
guessing is because it's trying to flush (syncfs?) imediately and
constantly.

You guys spend all day in the IO subsystem, any idea where I can keep
looking? It has persisted across reboots and kernel updates.

Re: Off-Topic Write cache disabling?

[Roger Heflin]

In general if you completely disable write caching things will likely
be unusable.    When I have benchmarked things attempting to turn off
write cache performance has been mostly unusable.

I usually change /etc/sysctl.conf and change these entries to control
the amount of dirty cache.

vm.dirty_background_bytes = 25000000
vm.dirty_bytes = 50000000

That says max dirty allowed is 50mb, and it syncs down to 25mb when it syncs.

If you want to tighten things up I would lower those numbers down to
say 10mb/5mb and that should limit how much you can have in the dirty
cache.

You can obviously go lower, but I would expect issues if you get too
low, but I don't know what too low is.

On Tue, Feb 3, 2015 at 11:45 AM, Weedy <weedy2887@gmail.com> wrote:
> Is there a kernel option or sysfs toggle that disables write caching?
> Or forces the kernel to commit everything constantly.
>
> ------
> I don't really want to join another ML, especially a higher traffic
> one just to ask this when it only bugs me sometimes. But I'll shut up
> if this is unwanted.
> ------
>
> I use a similar kernel config with respect to selected options on all
> my systems but this only effect my laptop.
> On any of my personal system or system I have remote access too
> nr_dirty drifts up and down and nr_writeback stays around 0 (assuming
> the system isn't working hard).
> On my laptop both nr_dirty and nr_writeback stay at 0. I can make them
> go up to 10ish if I untar something but then almost immediately go
> back to 0. If I didn't know better I would swear dirty_*_centisecs or
> something was set to a near instant commit interval but I haven't
> found evidence of that. The hard drive light blinks almost constantly
> once a second, even if I'm at a X login screen.
> As I said this doesn't bug me most of the time but if I let my FF
> session get too large or start multiple VMs anything that might make
> me swap a little, the machine pretty much dies from IOWAIT. Which I'm
> guessing is because it's trying to flush (syncfs?) imediately and
> constantly.
>
> You guys spend all day in the IO subsystem, any idea where I can keep
> looking? It has persisted across reboots and kernel updates.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Off-Topic Write cache disabling?

[Alireza Haghdoost]

> and it syncs down to 25mb when it syncs.

What do you mean by "syncs down to 25mb" ?

Re: Off-Topic Write cache disabling?

[Roger Heflin]

When dirty_cache hits 50mb it writes enough out to get the dirty_cache
down to 25MB (since 25000000 is the lower setting)

On Tue, Feb 3, 2015 at 1:16 PM, Alireza Haghdoost <alireza@cs.umn.edu> wrote:
>> and it syncs down to 25mb when it syncs.
>
> What do you mean by "syncs down to 25mb" ?

Re: Off-Topic Write cache disabling?

[Weedy]

On Tue, Feb 3, 2015 at 1:49 PM, Roger Heflin <rogerheflin@gmail.com> wrote:
> In general if you completely disable write caching things will likely
> be unusable.    When I have benchmarked things attempting to turn off
> write cache performance has been mostly unusable.


This has little to do with the problem I wanted help with.

Re: Off-Topic Write cache disabling?

[Bernd Schubert]

On 02/03/2015 07:49 PM, Roger Heflin wrote:
> In general if you completely disable write caching things will likely
> be unusable.    When I have benchmarked things attempting to turn off
> write cache performance has been mostly unusable.
> 
> I usually change /etc/sysctl.conf and change these entries to control
> the amount of dirty cache.
> 
> vm.dirty_background_bytes = 25000000
> vm.dirty_bytes = 50000000
> 
> That says max dirty allowed is 50mb, and it syncs down to 25mb when it syncs.

Er, where do you have this from? 
dirty_background_bytes and dirty_bytes are more about non-blocking and 
blocking dirty writes. And about the amount of dirty data that triggers 
it (i.e. there is no need to flush temporary file data to disk if files
get immediately deleted after writing them)
 
linux/Documentation/sysctl/vm.txt

==============================================================

dirty_background_bytes

Contains the amount of dirty memory at which the background kernel
flusher threads will start writeback.

Note: dirty_background_bytes is the counterpart of dirty_background_ratio. Only
one of them may be specified at a time. When one sysctl is written it is
immediately taken into account to evaluate the dirty memory limits and the
other appears as 0 when read.

==============================================================


==============================================================

dirty_bytes

Contains the amount of dirty memory at which a process generating disk writes
will itself start writeback.

Note: dirty_bytes is the counterpart of dirty_ratio. Only one of them may be
specified at a time. When one sysctl is written it is immediately taken into
account to evaluate the dirty memory limits and the other appears as 0 when
read.

Note: the minimum value allowed for dirty_bytes is two pages (in bytes); any
value lower than this limit will be ignored and the old configuration will be
retained.

==============================================================



Bernd

Re: Off-Topic Write cache disabling?

[Bernd Schubert]

On 02/03/2015 06:45 PM, Weedy wrote:
> Is there a kernel option or sysfs toggle that disables write caching?
> Or forces the kernel to commit everything constantly.
> 
> ------
> I don't really want to join another ML, especially a higher traffic
> one just to ask this when it only bugs me sometimes. But I'll shut up
> if this is unwanted.
> ------
> 
> I use a similar kernel config with respect to selected options on all
> my systems but this only effect my laptop.
> On any of my personal system or system I have remote access too
> nr_dirty drifts up and down and nr_writeback stays around 0 (assuming
> the system isn't working hard).
> On my laptop both nr_dirty and nr_writeback stay at 0. I can make them
> go up to 10ish if I untar something but then almost immediately go
> back to 0. If I didn't know better I would swear dirty_*_centisecs or
> something was set to a near instant commit interval but I haven't
> found evidence of that. The hard drive light blinks almost constantly
> once a second, even if I'm at a X login screen.
> As I said this doesn't bug me most of the time but if I let my FF
> session get too large or start multiple VMs anything that might make
> me swap a little, the machine pretty much dies from IOWAIT. Which I'm
> guessing is because it's trying to flush (syncfs?) imediately and
> constantly.
> 
> You guys spend all day in the IO subsystem, any idea where I can keep
> looking? It has persisted across reboots and kernel updates.


Try mount -osync or force your application to write with direct IO.


Cheers,
Bernd

Re: Off-Topic Write cache disabling?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2389 bytes --]

On Tue, 3 Feb 2015 12:45:44 -0500 Weedy <weedy2887@gmail.com> wrote:

> Is there a kernel option or sysfs toggle that disables write caching?
> Or forces the kernel to commit everything constantly.
> 
> ------
> I don't really want to join another ML, especially a higher traffic
> one just to ask this when it only bugs me sometimes. But I'll shut up
> if this is unwanted.
> ------
> 
> I use a similar kernel config with respect to selected options on all
> my systems but this only effect my laptop.

Maybe it has something to do with "laptop_mode"
  
    /proc/sys/vm/laptop_mode
    https://www.kernel.org/doc/Documentation/laptops/laptop-mode.txt

> On any of my personal system or system I have remote access too
> nr_dirty drifts up and down and nr_writeback stays around 0 (assuming
> the system isn't working hard).
> On my laptop both nr_dirty and nr_writeback stay at 0. I can make them
> go up to 10ish if I untar something but then almost immediately go
> back to 0. If I didn't know better I would swear dirty_*_centisecs or
> something was set to a near instant commit interval but I haven't
> found evidence of that. The hard drive light blinks almost constantly
> once a second, even if I'm at a X login screen.

That doesn't quite sound like laptop mode...

laptop mode syncs things more aggressively when the disk is spinning, and
more lazily when it isn't ... or something like that.

So I can imagine nr_dirty staying low - any read will spin the disk, and that
will quickly flush out any dirty pages.
The once-a-second blink is harder to explain.

Still, it would be worth checking if laptop mode is enabled, and if so
disable it and see if anything changes.

NeilBrown



> As I said this doesn't bug me most of the time but if I let my FF
> session get too large or start multiple VMs anything that might make
> me swap a little, the machine pretty much dies from IOWAIT. Which I'm
> guessing is because it's trying to flush (syncfs?) imediately and
> constantly.
> 
> You guys spend all day in the IO subsystem, any idea where I can keep
> looking? It has persisted across reboots and kernel updates.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]