linux disk access when idle

linux disk access when idle

[Matt Garman]

The scope of this question is probably broader than Linux RAID, but
I thought I'd try here before going to the LKML...

I have a fileserver/NAS box that has four Western Digital GreenPower
drives in software RAID-5 (i.e. Linux md).  There was some noise
about these drives not too long ago: when used under Linux, they
tend to have really high (and rapidly increasing) SMART
Load_Cycle_Count values[1].

Basically, in order to save power, these drives park the read/write
heads after so many seconds of inactivity.  According to
SilentPCReview[2], the 1 TB model uses 5.7W when idle and only 3.7W
when the heads are unloaded.  Apparantly, each head parking event
causes teh Load_Cycle_Count value to increase.

Most people seek to disable this head parking behavior to stop that
Load_Cycle_Count value from growing alarmingly high.  I actually
want to use it to it's full potential.

I've got a Kill-A-Watt electricity meter connected to my NAS box.  I
see a difference of 10 watts power usage when the heads park (I can
hear them unload, so I know when it happens).  Allowing for some
power supply inefficiency, I'd say my observation is consistent with
SPCR's numbers.

The problem is, the drives don't stay in this parked state very
long.  I haven't actually timed the state changes, but average power
consumption over a long time (e.g. a week or more) is at the higher
(i.e. +10) level.

So I figure, something is accessing these drives shortly after the
heads park, causing them to un-park (and increasing power
consumption).  But this machine is idle 95% of the time.  And even
then, the overwhelming majority of the accesses are reads, with very
few writes (literally, a handful of writes per week).

So what I'm trying to figure out is, what is causing the disk
access?  It could be any one of:

    - Kernel
    - RAID subsystem (i.e. md)
    - XFS filesystem
    - NFS
    - Samba
    - ???

I'm hoping someone has enough knowledge of these systems to point me
in the right direction for tuning things.  The goal is that when the
machine is idle, it is "truly" idle, meaning, no disk accesses take
place and the heads can stay parked (thus saving energy).

Thank you,
Matt

LINKS:
[1] http://groups.google.com/group/linux.kernel/browse_thread/thread/505ccf760023d132/7e4f4e996f911efd

[2] http://www.silentpcreview.com/article804-page2.html

Re: linux disk access when idle

[Richard Scobie]

Matt Garman wrote:

> So what I'm trying to figure out is, what is causing the disk
> access?  It could be any one of:
> 
>     - Kernel
>     - RAID subsystem (i.e. md)
>     - XFS filesystem
>     - NFS
>     - Samba
>     - ???

Are you running smartd? It polls the drives every 30minutes.

Regards,

Richard

RE: linux disk access when idle

[Henry, Andrew]

I had a similar issue, as I wanted to take full advantage of parking heads but they kept coming alive again.  I was putting them to sleep with hdparm.

I gave up at the end, but thought I had read that it was likely syslog or other internals that were polling the disc too frequently.  Sorry Im a bit vague on the details.

If this was the case (I didn't really investigate) then I thought, stuff it, because I wanted syslog to keep doing it's thing.

--andrew

RE: linux disk access when idle

[Jon Hardcastle]

--- On Fri, 21/8/09, Henry, Andrew <andrew.henry@logica.com> wrote:

> From: Henry, Andrew <andrew.henry@logica.com>
> Subject: RE: linux disk access when idle
> To: "Matt Garman" <matthew.garman@gmail.com>, "linux-raid" <linux-raid@vger.kernel.org>
> Date: Friday, 21 August, 2009, 7:42 AM
> I had a similar issue, as I wanted to
> take full advantage of parking heads but they kept coming
> alive again.  I was putting them to sleep with hdparm.
> 
> I gave up at the end, but thought I had read that it was
> likely syslog or other internals that were polling the disc
> too frequently.  Sorry Im a bit vague on the details.
> 
> If this was the case (I didn't really investigate) then I
> thought, stuff it, because I wanted syslog to keep doing
> it's thing.
> 
> --andrew
> 
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

I did this with the data drives in my array - never managed it with the system drives though.

You need to tell smartd to not poll if the drives are asleep and also tell syslog not to poll too often. 

There is some for of flag you can apply to see write requests and the application that is doing them.......?

BUT a curious problem I have - and have NEVER really solved is that when a drive is asleep if a scheduled smart check runs - the drive doesn't wake up quickly enough so smart brands the drive as 'incapable of a smart test' and sends me a warning email.... every single time...

-----------------------
N: Jon Hardcastle
E: Jon@eHardcastle.com
'Do not worry about tomorrow, for tomorrow will bring worries of its own.'
-----------------------


      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: linux disk access when idle

[Foster_Brian]

> There is some for of flag you can apply to see write requests and the
> application that is doing them.......?
> 

You might find this useful:

echo 1 > /proc/sys/vm/block_dump

Then watch your syslog files...

Brian

Re: linux disk access when idle

[Matt Garman]

On Fri, Aug 21, 2009 at 06:45:45AM +1200, Richard Scobie wrote:
> Matt Garman wrote:
>> So what I'm trying to figure out is, what is causing the disk
>> access?  It could be any one of:
>>
>>     - Kernel
>>     - RAID subsystem (i.e. md)
>>     - XFS filesystem
>>     - NFS
>>     - Samba
>>     - ???
>
> Are you running smartd? It polls the drives every 30minutes.

I was running smartd, until I received your post :)

Unfortunately, this does not help keep the heads parked any longer.

Nonetheless, thank you for the hint.

Matt

Re: linux disk access when idle

[Ric Wheeler]

On 08/21/2009 12:59 PM, Matt Garman wrote:
> On Fri, Aug 21, 2009 at 06:45:45AM +1200, Richard Scobie wrote:
>    
>> Matt Garman wrote:
>>      
>>> So what I'm trying to figure out is, what is causing the disk
>>> access?  It could be any one of:
>>>
>>>      - Kernel
>>>      - RAID subsystem (i.e. md)
>>>      - XFS filesystem
>>>      - NFS
>>>      - Samba
>>>      - ???
>>>        
>> Are you running smartd? It polls the drives every 30minutes.
>>      
> I was running smartd, until I received your post :)
>
> Unfortunately, this does not help keep the heads parked any longer.
>
> Nonetheless, thank you for the hint.
>
> Matt
>
>
>    
I think that you can use blktrace to see who is issuing the IO's that 
spin up your drives....

ric

Re: linux disk access when idle

[Matt Garman]

On Fri, Aug 21, 2009 at 08:42:09AM +0200, Henry, Andrew wrote:
> I had a similar issue, as I wanted to take full advantage of
> parking heads but they kept coming alive again.  I was putting
> them to sleep with hdparm.
> 
> I gave up at the end, but thought I had read that it was likely
> syslog or other internals that were polling the disc too
> frequently.  Sorry Im a bit vague on the details.
> 
> If this was the case (I didn't really investigate) then I thought,
> stuff it, because I wanted syslog to keep doing it's thing.

Good to know.  I should have mentioned, however, that my RAID5
drives are just for data storage.  The OS (and therefore syslog and
friends) is actually on a compact flash card (using one of those
compact flash to IDE converters).

Thanks,
Matt

Re: linux disk access when idle

[Matt Garman]

On Fri, Aug 21, 2009 at 12:35:19PM -0400, Foster_Brian@emc.com wrote:
> > There is some for of flag you can apply to see write requests and the
> > application that is doing them.......?
> > 
> 
> You might find this useful:
> 
> echo 1 > /proc/sys/vm/block_dump
> 
> Then watch your syslog files...

Thank you.  That along with Ric Wheeler's suggestion of blktrace
ought to help me get to the bottom of this.

Thanks again!
Matt

Re: linux disk access when idle

[Bill Davidsen]

Matt Garman wrote:
> The scope of this question is probably broader than Linux RAID, but
> I thought I'd try here before going to the LKML...
>
> I have a fileserver/NAS box that has four Western Digital GreenPower
> drives in software RAID-5 (i.e. Linux md).  There was some noise
> about these drives not too long ago: when used under Linux, they
> tend to have really high (and rapidly increasing) SMART
> Load_Cycle_Count values[1].
>
> Basically, in order to save power, these drives park the read/write
> heads after so many seconds of inactivity.  According to
> SilentPCReview[2], the 1 TB model uses 5.7W when idle and only 3.7W
> when the heads are unloaded.  Apparantly, each head parking event
> causes teh Load_Cycle_Count value to increase.
>
> Most people seek to disable this head parking behavior to stop that
> Load_Cycle_Count value from growing alarmingly high.  I actually
> want to use it to it's full potential.
>
> I've got a Kill-A-Watt electricity meter connected to my NAS box.  I
> see a difference of 10 watts power usage when the heads park (I can
> hear them unload, so I know when it happens).  Allowing for some
> power supply inefficiency, I'd say my observation is consistent with
> SPCR's numbers.
>
> The problem is, the drives don't stay in this parked state very
> long.  I haven't actually timed the state changes, but average power
> consumption over a long time (e.g. a week or more) is at the higher
> (i.e. +10) level.
>
> So I figure, something is accessing these drives shortly after the
> heads park, causing them to un-park (and increasing power
> consumption).  But this machine is idle 95% of the time.  And even
> then, the overwhelming majority of the accesses are reads, with very
> few writes (literally, a handful of writes per week).
>
>   
/proc/sys/vm/dirty_expire_centisecs still set to 30 sec?

> So what I'm trying to figure out is, what is causing the disk
> access?  It could be any one of:
>
>     - Kernel
>     - RAID subsystem (i.e. md)
>     - XFS filesystem
>     - NFS
>     - Samba
>     - ???
>
> I'm hoping someone has enough knowledge of these systems to point me
> in the right direction for tuning things.  The goal is that when the
> machine is idle, it is "truly" idle, meaning, no disk accesses take
> place and the heads can stay parked (thus saving energy).
>
> Thank you,
> Matt
>
> LINKS:
> [1] http://groups.google.com/group/linux.kernel/browse_thread/thread/505ccf760023d132/7e4f4e996f911efd
>
> [2] http://www.silentpcreview.com/article804-page2.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   


-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money back."
    - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.

Re: linux disk access when idle

[Matt Garman]

On Fri, Aug 21, 2009 at 06:00:43PM -0400, Bill Davidsen wrote:
> /proc/sys/vm/dirty_expire_centisecs still set to 30 sec?

$ cat /proc/sys/vm/dirty_expire_centisecs
2999

Yup.  Maybe I don't understand exactly what this does, but if the
drives are idle (i.e. no reads or writes) for say an hour or more,
why would there be any dirty data to flush?

That's why I'm confused---this machine literally goes for hours at a
time without and read or write attemps on these drives.

-Matt

Re: linux disk access when idle

[Carl A. Cook]

I haven't followed this whole thread, but there is some aspect of ext3 journalling which hits that disk with regularity.  Obviously not a write flush, but some sort of check out of insecurity or something.


On Friday 21 August 2009 16:10:39 Matt Garman wrote:
> On Fri, Aug 21, 2009 at 06:00:43PM -0400, Bill Davidsen wrote:
> > /proc/sys/vm/dirty_expire_centisecs still set to 30 sec?
> 
> $ cat /proc/sys/vm/dirty_expire_centisecs
> 2999
> 
> Yup.  Maybe I don't understand exactly what this does, but if the
> drives are idle (i.e. no reads or writes) for say an hour or more,
> why would there be any dirty data to flush?
> 
> That's why I'm confused---this machine literally goes for hours at a
> time without and read or write attemps on these drives.
> 
> -Matt
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: linux disk access when idle

[berk walker]

I thought it was just a trait of the animal.  My Linux systems have 
always done disk access every few seconds, 24/7. From '92 on.
b-

Re: linux disk access when idle

[John Robinson]

On 22/08/2009 02:41, berk walker wrote:
> I thought it was just a trait of the animal.  My Linux systems have 
> always done disk access every few seconds, 24/7. From '92 on.

Not necessarily. I don't know what they did to achieve it, but Netgear's 
ReadyNAS devices can be configured to park the drives when they're left 
alone, it works (unless you're using SqueezeCenter), and they're 
Linux-based.

Cheers,

John.

Re: linux disk access when idle

[Richard Scobie]

John Robinson wrote:

> Not necessarily. I don't know what they did to achieve it, but Netgear's 
> ReadyNAS devices can be configured to park the drives when they're left 
> alone, it works (unless you're using SqueezeCenter), and they're 
> Linux-based.

As I understand it, the ReadyNAS uses XFS.

Regards,

Richard

Re: linux disk access when idle

[Kyle Liddell]

On Thu, 2009-08-20 at 11:35 -0500, Matt Garman wrote:
> So what I'm trying to figure out is, what is causing the disk
> access?  It could be any one of:
> 
>     - Kernel
>     - RAID subsystem (i.e. md)
>     - XFS filesystem
>     - NFS
>     - Samba
>     - ???

For one, syslogd.  
It's probably worthwhile to look up information for laptop power saving.
In particular, laptop-mode in the kernel, and laptop-mode-tools:
http://samwel.tk/laptop_mode
I use this on my laptop, and I don't think I ever have a "surprise" disk
spinup.  The whole laptop-mode-tools setup is probably much more than
you want, but it's a good place to start.  The FAQ there also mentions
how to get syslog to stop syncing to disk.  
You may also find 'lm-profiler' useful - this will tell you the which
process is causing disk accesses.  Finally, for general power saving,
'powertop' tracks the power state information for your CPU, and will
list the processes that are causing your CPU to come out of any
sleep/idle states.

Re: linux disk access when idle

[Tapani Tarvainen]

Just a data point: I've got a box (Debian Lenny) with five disks,
system disk plus 4-disk RAID5 array, used as a backup server
(rsnapshot), and I power the disks in the raid array down after
running the backup - and they stay powered down, even though I don't
umount the filesystems let alone stop the array. When accessed (like
for restoring something from the backup), they wake up, so I've got an
hourly cron job powering them down again - but most days (when
there're no restores) they are powered up only during the backup run.

So it can be done. It's not filesystem type dependent either, there's
one jfs and one ext3 in the array.

The system disk stays up, though. Some experimenting suggested
that getting it to stay down would require putting /tmp and parts
of /var on a ramdisk (/var/log at least), but I didn't go to
the trouble of tracking down all disk-awakening services.

-- 
Tapani Tarvainen

Re: linux disk access when idle

[Goswin von Brederlow]

Matt Garman <matthew.garman@gmail.com> writes:

> On Fri, Aug 21, 2009 at 06:45:45AM +1200, Richard Scobie wrote:
>> Matt Garman wrote:
>>> So what I'm trying to figure out is, what is causing the disk
>>> access?  It could be any one of:
>>>
>>>     - Kernel
>>>     - RAID subsystem (i.e. md)
>>>     - XFS filesystem
>>>     - NFS
>>>     - Samba
>>>     - ???
>>
>> Are you running smartd? It polls the drives every 30minutes.
>
> I was running smartd, until I received your post :)
>
> Unfortunately, this does not help keep the heads parked any longer.
>
> Nonetheless, thank you for the hint.
>
> Matt

Smartd has an option not to wake up disks. If you have smartd scann
less often than the suspend time of the disk then idle disk can spinn
down and stay that way.

MfG
        Goswin

Re: linux disk access when idle

[Goswin von Brederlow]

Tapani Tarvainen <raid@tapanitarvainen.fi> writes:

> Just a data point: I've got a box (Debian Lenny) with five disks,
> system disk plus 4-disk RAID5 array, used as a backup server
> (rsnapshot), and I power the disks in the raid array down after
> running the backup - and they stay powered down, even though I don't
> umount the filesystems let alone stop the array. When accessed (like
> for restoring something from the backup), they wake up, so I've got an
> hourly cron job powering them down again - but most days (when
> there're no restores) they are powered up only during the backup run.
>
> So it can be done. It's not filesystem type dependent either, there's
> one jfs and one ext3 in the array.
>
> The system disk stays up, though. Some experimenting suggested
> that getting it to stay down would require putting /tmp and parts
> of /var on a ramdisk (/var/log at least), but I didn't go to
> the trouble of tracking down all disk-awakening services.

Same here. I have a raid1 for the system on 2 drives and a raid5 over
6 drives for data. The raid5 is ext3 powers down after 15 minutes
without access. The raid1 stays up all the time due to /var/log/syslog
being written to too often. /tmp is tmpfs though.

MfG
        Goswin

Re: linux disk access when idle

[Matt Garman]

On Fri, Aug 21, 2009 at 01:01:48PM -0400, Ric Wheeler wrote:
> On 08/21/2009 12:59 PM, Matt Garman wrote:
>>>> So what I'm trying to figure out is, what is causing the disk
>>>> access?  It could be any one of:
>>>>
>>>>      - Kernel
>>>>      - RAID subsystem (i.e. md)
>>>>      - XFS filesystem
>>>>      - NFS
>>>>      - Samba
>>>>      - ???
>>>>        
>>> Are you running smartd? It polls the drives every 30minutes.
>>    
> I think that you can use blktrace to see who is issuing the IO's
> that  spin up your drives....

For five hours today, I ran blktrace on md1 as well as the drives
that make up md1 (sd[a-d]).  I also enabled block dump via

    echo 1 > /proc/sys/vm/block_dump

Nothing.  Neither block_dump nor blktrace recorded anything on md1
or sd[abcd].  I caught one instance where the heads parked (I both
heard them park as well as saw the power meter show the 10 watt
drop).  They only stayed parked for about five minutes.  Yet neither
blktrace nor block_dump caught anything.

I'm going to try lm-profiler now (from the laptop-mode-tools
package), but I have low expectations.

I'm beginning to think that it's not even direct access to the
drives that cause the heads to unpark, e.g. perhaps even activity on
my system drive (a compact flash card connected to the PATA port)
prevents the data drives from parking their heads.

Re: linux disk access when idle

[Bill Davidsen]

Matt Garman wrote:
> On Fri, Aug 21, 2009 at 01:01:48PM -0400, Ric Wheeler wrote:
>   
>> On 08/21/2009 12:59 PM, Matt Garman wrote:
>>     
>>>>> So what I'm trying to figure out is, what is causing the disk
>>>>> access?  It could be any one of:
>>>>>
>>>>>      - Kernel
>>>>>      - RAID subsystem (i.e. md)
>>>>>      - XFS filesystem
>>>>>      - NFS
>>>>>      - Samba
>>>>>      - ???
>>>>>        
>>>>>           
>>>> Are you running smartd? It polls the drives every 30minutes.
>>>>         
>>>    
>>>       
>> I think that you can use blktrace to see who is issuing the IO's
>> that  spin up your drives....
>>     
>
> For five hours today, I ran blktrace on md1 as well as the drives
> that make up md1 (sd[a-d]).  I also enabled block dump via
>
>     echo 1 > /proc/sys/vm/block_dump
>
> Nothing.  Neither block_dump nor blktrace recorded anything on md1
> or sd[abcd].  I caught one instance where the heads parked (I both
> heard them park as well as saw the power meter show the 10 watt
> drop).  They only stayed parked for about five minutes.  Yet neither
> blktrace nor block_dump caught anything.
>
> I'm going to try lm-profiler now (from the laptop-mode-tools
> package), but I have low expectations.
>
> I'm beginning to think that it's not even direct access to the
> drives that cause the heads to unpark, e.g. perhaps even activity on
> my system drive (a compact flash card connected to the PATA port)
> prevents the data drives from parking their heads.
>   

Are you doing regular self test? I'm not sure if non-io connections 
would show using those tools, I just haven't tried them at that level.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc

"Now we have another quarterback besides Kurt Warner telling us during postgame
interviews that he owes every great thing that happens to him on a football
field to his faith in Jesus. I knew there had to be a reason why the Almighty
included a mute button on my remote."
			-- Arthur Troyer on Tim Tebow (Sports Illustrated)

Re: linux disk access when idle

[Billy Crook]

On Sat, Aug 22, 2009 at 00:04, Tapani Tarvainen<raid@tapanitarvainen.fi> wrote:
> so I've got an
> hourly cron job powering them down again

What's in your cronjob?

Re: linux disk access when idle

[Matt Garman]

On Sat, Aug 22, 2009 at 04:04:43PM -0500, Matt Garman wrote:
> I'm beginning to think that it's not even direct access to the
> drives that cause the heads to unpark, e.g. perhaps even activity
> on my system drive (a compact flash card connected to the PATA
> port) prevents the data drives from parking their heads.

This is correct.  One thing I noticed yesterday, when I was running
blktrace and having block_dump enabled: the heads *never* parked
during this time.  Even though I had zero accesses on the data
drives, the system drive was getting hit pretty heavily (because I
deliberately had syslog write the block_dump stuff).  When I finally
stopped all the heavy logging, I ran "sync" and noticed that a few
seconds later, the heads parked.

Almost exactly five minutes later, the heads un-parked.

I repeated: I ran the "sync" command, the heads unloaded, five
minutes later, they re-loaded.  I repeated this exactly several
times.

So I re-enabled block_dump (i.e. "echo 1 > /proc/sys/vm/block_dump"), 
but disabled syslog (so the system drive wouldn't be thrashed).  I
ran the "sync, then wait five minutes test": pdflush and kjournald
apparently caused the wakeup after five minutes.

I set my system drive to ext2 (instead of ext3) in /etc/fstab and
rebooted.  I was under the impression that using ext2 instead of
ext3 would eliminate the need for kjournald, but this doesn't appear
to be the case.

So, based on what I've seen so far, there's some "five minute thing"
going on that involves pdflush and/or kjournald.

-Matt