* Question on blocks periodic writes
@ 2016-11-09 17:20 Theophanis Kontogiannis
2016-11-10 2:00 ` NeilBrown
0 siblings, 1 reply; 6+ messages in thread
From: Theophanis Kontogiannis @ 2016-11-09 17:20 UTC (permalink / raw)
To: Linux RAID
Hello All,
I am in the middle of bringing my server's power consumption to an
absolute minimum.
Have already reduced idle power from 110W to 52W (including 10W
consumed by the UPS).
Having enabled spin down to all 5 x 2TB disks, with 5 seconds idle
times, I noticed that the disks wake up quite often without the server
doing anything actual.
Following
'echo 1 > /proc/sys/vm/block_dump'
I can not miss that
'dmesg -c'
reports frequent messages like:
[ 8662.496150] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
[ 8662.496185] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors)
[ 8662.496253] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors)
[ 8662.496269] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors)
[ 8662.496282] md0_raid6(631): WRITE block 16 on sde3 (8 sectors)
[ 8664.849252] md0_raid6(631): WRITE block 8 on sda3 (1 sectors)
[ 8664.849276] md0_raid6(631): WRITE block 8 on sdf3 (1 sectors)
[ 8664.849287] md0_raid6(631): WRITE block 8 on sdg3 (1 sectors)
[ 8664.849298] md0_raid6(631): WRITE block 8 on sdd3 (1 sectors)
[ 8664.849352] md0_raid6(631): WRITE block 8 on sde3 (1 sectors)
[ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors)
[ 8664.902688] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
[ 8664.902705] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors)
[ 8664.902715] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors)
[ 8664.902725] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors)
[ 8664.902735] md0_raid6(631): WRITE block 16 on sde3 (8 sectors)
[ 8665.100056] md1_raid6(630): WRITE block 8 on sda1 (1 sectors)
[ 8665.100107] md1_raid6(630): WRITE block 8 on sdf1 (1 sectors)
[ 8665.100164] md1_raid6(630): WRITE block 8 on sde1 (1 sectors)
[ 8665.100217] md1_raid6(630): WRITE block 8 on sdg1 (1 sectors)
[ 8665.100467] md1_raid6(630): WRITE block 8 on sdd1 (1 sectors)
[ 8665.177050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
[ 8665.177098] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors)
[ 8665.177154] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors)
[ 8665.177207] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors)
[ 8665.177431] md0_raid6(631): WRITE block 16 on sde3 (8 sectors)
[ 8665.225978] md0_raid6(631): WRITE block 8 on sda3 (1 sectors)
[ 8665.225996] md0_raid6(631): WRITE block 8 on sdf3 (1 sectors)
[ 8665.226064] md0_raid6(631): WRITE block 8 on sdg3 (1 sectors)
[ 8665.226111] md0_raid6(631): WRITE block 8 on sdd3 (1 sectors)
[ 8665.226191] md0_raid6(631): WRITE block 8 on sde3 (1 sectors)
[ 8670.269138] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
[ 8670.269180] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors)
[ 8670.269237] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors)
[ 8670.269291] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors)
[ 8670.269344] md0_raid6(631): WRITE block 16 on sde3 (8 sectors)
[ 8680.269050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
[ 8680.269092] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors)
[ 8680.269147] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors)
[ 8680.269249] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors)
[ 8680.269428] md0_raid6(631): WRITE block 16 on sde3 (8 sectors)
I guess those messages are the reason for the frequent disks spin up.
What is the reason behind those writes?
Can I affect it? Should I touch it?
---
Best regards,
ΜΦΧ,
Theophanis Kontogiannis
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes
2016-11-09 17:20 Question on blocks periodic writes Theophanis Kontogiannis
@ 2016-11-10 2:00 ` NeilBrown
2016-11-10 16:10 ` Wols Lists
0 siblings, 1 reply; 6+ messages in thread
From: NeilBrown @ 2016-11-10 2:00 UTC (permalink / raw)
To: Theophanis Kontogiannis, Linux RAID
[-- Attachment #1: Type: text/plain, Size: 2261 bytes --]
On Thu, Nov 10 2016, Theophanis Kontogiannis wrote:
> Hello All,
>
> I am in the middle of bringing my server's power consumption to an
> absolute minimum.
>
> Have already reduced idle power from 110W to 52W (including 10W
> consumed by the UPS).
>
> Having enabled spin down to all 5 x 2TB disks, with 5 seconds idle
> times, I noticed that the disks wake up quite often without the server
> doing anything actual.
>
> Following
>
> 'echo 1 > /proc/sys/vm/block_dump'
>
> I can not miss that
>
> 'dmesg -c'
>
> reports frequent messages like:
>
> [ 8662.496150] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
> [ 8662.496185] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors)
> [ 8662.496253] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors)
> [ 8662.496269] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors)
> [ 8662.496282] md0_raid6(631): WRITE block 16 on sde3 (8 sectors)
These are probably the md bitmap being updated.
If you provided some basic detail about you array like
mdadm --detail /dev/md1
mdadm --examine /dev/sda3
it would be easier to be sure.
> [ 8664.849252] md0_raid6(631): WRITE block 8 on sda3 (1 sectors)
> [ 8664.849276] md0_raid6(631): WRITE block 8 on sdf3 (1 sectors)
> [ 8664.849287] md0_raid6(631): WRITE block 8 on sdg3 (1 sectors)
> [ 8664.849298] md0_raid6(631): WRITE block 8 on sdd3 (1 sectors)
> [ 8664.849352] md0_raid6(631): WRITE block 8 on sde3 (1 sectors)
This is probably the superblock being updated.
> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors)
This is XFS doing something. md cannot possibly stop all IO while the
filesystem performs occasional IO. If these continue, you need to
discuss with xfs developers how to stop it. If the writes to individual
drives continue after there are no writes to 'md1', then it is worth
coming back here to ask.
> [ 8664.902688] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
> [ 8665.177050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
> [ 8670.269138] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
> [ 8680.269050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors)
The delay here is 270ms, then 5 seconds, then 10 seconds.
Does it reach a stable state? What is the period in the stable state?
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes
2016-11-10 2:00 ` NeilBrown
@ 2016-11-10 16:10 ` Wols Lists
2016-11-11 1:52 ` NeilBrown
0 siblings, 1 reply; 6+ messages in thread
From: Wols Lists @ 2016-11-10 16:10 UTC (permalink / raw)
To: NeilBrown, Theophanis Kontogiannis, Linux RAID
On 10/11/16 02:00, NeilBrown wrote:
>> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors)
> This is XFS doing something. md cannot possibly stop all IO while the
> filesystem performs occasional IO. If these continue, you need to
> discuss with xfs developers how to stop it. If the writes to individual
> drives continue after there are no writes to 'md1', then it is worth
> coming back here to ask.
>
>
Would the new journal feature be any help?
I haven't dug in enough to understand it properly, and it would increase
the vulnerability of the system to a journal failure, but the feature
itself seems almost perfect for batching writes and enabling the disks
to spin down for extended periods.
Cheers,
Wol
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes
2016-11-10 16:10 ` Wols Lists
@ 2016-11-11 1:52 ` NeilBrown
2016-11-11 12:11 ` Theophanis Kontogiannis
2016-11-11 12:13 ` Theophanis Kontogiannis
0 siblings, 2 replies; 6+ messages in thread
From: NeilBrown @ 2016-11-11 1:52 UTC (permalink / raw)
To: Wols Lists, Theophanis Kontogiannis, Linux RAID
[-- Attachment #1: Type: text/plain, Size: 1070 bytes --]
On Fri, Nov 11 2016, Wols Lists wrote:
> On 10/11/16 02:00, NeilBrown wrote:
>>> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors)
>> This is XFS doing something. md cannot possibly stop all IO while the
>> filesystem performs occasional IO. If these continue, you need to
>> discuss with xfs developers how to stop it. If the writes to individual
>> drives continue after there are no writes to 'md1', then it is worth
>> coming back here to ask.
>>
>>
> Would the new journal feature be any help?
Probably not, though until we know what is causing the writes, it is
hard to say.
>
> I haven't dug in enough to understand it properly, and it would increase
> the vulnerability of the system to a journal failure, but the feature
> itself seems almost perfect for batching writes and enabling the disks
> to spin down for extended periods.
You might be able to build functionality onto the journal which allows
the drives in the main array to stay idle for longer, but it doesn't try
to do that at present.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes
2016-11-11 1:52 ` NeilBrown
@ 2016-11-11 12:11 ` Theophanis Kontogiannis
2016-11-11 12:13 ` Theophanis Kontogiannis
1 sibling, 0 replies; 6+ messages in thread
From: Theophanis Kontogiannis @ 2016-11-11 12:11 UTC (permalink / raw)
To: NeilBrown; +Cc: Wols Lists, Linux RAID
Hi Neil,
Hi Community,
Regarding XFS we can ignore it. That file system will be moved to an
ext2 file system on a CF.
So we are left with the rest:
-bash-4.2# mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Thu Jun 16 18:02:57 2016
Raid Level : raid6
Array Size : 9397248 (8.96 GiB 9.62 GB)
Used Dev Size : 3132416 (2.99 GiB 3.21 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Fri Nov 11 14:05:33 2016
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : tweety.example.com:1 (local to host tweety.example.com)
UUID : 98e2af83:dc074310:d1639adb:3f19f0d3
Events : 127
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 49 1 active sync /dev/sdd1
2 8 65 2 active sync /dev/sde1
3 8 81 3 active sync /dev/sdf1
4 8 97 4 active sync /dev/sdg1
-bash-4.2# mdadm --examine /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 98e2af83:dc074310:d1639adb:3f19f0d3
Name : tweety.example.com:1 (local to host tweety.example.com)
Creation Time : Thu Jun 16 18:02:57 2016
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 6264832 (2.99 GiB 3.21 GB)
Array Size : 9397248 (8.96 GiB 9.62 GB)
Data Offset : 4096 sectors
Super Offset : 8 sectors
Unused Space : before=4008 sectors, after=0 sectors
State : clean
Device UUID : 5ff290a3:68faf9d0:22edd403:abbaf970
Update Time : Fri Nov 11 14:05:59 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 55812945 - correct
Events : 127
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
Thank you ALL
---
Best regards,
ΜΦΧ,
Theophanis Kontogiannis
On Fri, Nov 11, 2016 at 3:52 AM, NeilBrown <neilb@suse.com> wrote:
> On Fri, Nov 11 2016, Wols Lists wrote:
>
>> On 10/11/16 02:00, NeilBrown wrote:
>>>> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors)
>>> This is XFS doing something. md cannot possibly stop all IO while the
>>> filesystem performs occasional IO. If these continue, you need to
>>> discuss with xfs developers how to stop it. If the writes to individual
>>> drives continue after there are no writes to 'md1', then it is worth
>>> coming back here to ask.
>>>
>>>
>> Would the new journal feature be any help?
>
> Probably not, though until we know what is causing the writes, it is
> hard to say.
>
>>
>> I haven't dug in enough to understand it properly, and it would increase
>> the vulnerability of the system to a journal failure, but the feature
>> itself seems almost perfect for batching writes and enabling the disks
>> to spin down for extended periods.
>
> You might be able to build functionality onto the journal which allows
> the drives in the main array to stay idle for longer, but it doesn't try
> to do that at present.
>
> NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes
2016-11-11 1:52 ` NeilBrown
2016-11-11 12:11 ` Theophanis Kontogiannis
@ 2016-11-11 12:13 ` Theophanis Kontogiannis
1 sibling, 0 replies; 6+ messages in thread
From: Theophanis Kontogiannis @ 2016-11-11 12:13 UTC (permalink / raw)
To: NeilBrown; +Cc: Wols Lists, Linux RAID
Have also run the following for about 6 hours:
-bash-4.2# cat iotop-results
Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
PID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
696 be/3 root 0.00 B 956.00 K 0.00 % 0.31 % [jbd2/md0-8]
750 be/4 root 0.00 B 0.00 B 0.00 % 0.05 % smartd -n -q never
2134 be/4 mysql 624.00 K 344.85 M 0.00 % 0.01 % mysqld
--basedir=/usr --datadir=/mnt/md/md0/DBs/mys~riadb/mariadb.pid
--socket=/var/lib/mysql/mysql.sock
14110 be/4 root 0.00 B 0.00 B 0.00 % 0.01 % udisksd --no-debug
5217 be/4 root 0.00 B 0.00 B 0.00 % 0.29 % [kworker/1:0]
2454 be/4 root 0.00 B 0.00 B 0.00 % 0.04 % [kworker/0:2]
31764 be/4 root 4.00 K 0.00 B 0.00 % 0.00 % -bash
2368 be/4 root 0.00 B 0.00 B 0.00 % 0.01 % [kworker/1:3]
4584 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/1:1]
5265 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/0:0]
4595 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/0:1]
31747 be/4 root 0.00 B 8.00 K 0.00 % 0.00 % sshd: root@pts/3
761 be/4 root 0.00 B 10.20 M 0.00 % 0.00 % rsyslogd -n
7437 be/4 apache 0.00 B 24.00 K 0.00 % 0.00 % httpd -DFOREGROUND
14625 be/4 apache 0.00 B 20.00 K 0.00 % 0.00 % httpd -DFOREGROUND
10059 be/4 apache 0.00 B 8.00 K 0.00 % 0.00 % httpd -DFOREGROUND
10062 be/4 apache 0.00 B 24.00 K 0.00 % 0.00 % httpd -DFOREGROUND
10094 be/4 apache 0.00 B 16.00 K 0.00 % 0.00 % httpd -DFOREGROUND
10095 be/4 apache 0.00 B 20.00 K 0.00 % 0.00 % httpd -DFOREGROUND
10141 be/4 apache 0.00 B 24.00 K 0.00 % 0.00 % httpd -DFOREGROUND
12220 be/4 root 0.00 B 9.99 M 0.00 % 0.00 % python
/usr/bin/weewxd --daemon --pidfile=/var/run/weewx.pid
/etc/weewx/weewx.conf
---
Best regards,
ΜΦΧ,
Theophanis Kontogiannis
On Fri, Nov 11, 2016 at 3:52 AM, NeilBrown <neilb@suse.com> wrote:
> On Fri, Nov 11 2016, Wols Lists wrote:
>
>> On 10/11/16 02:00, NeilBrown wrote:
>>>> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors)
>>> This is XFS doing something. md cannot possibly stop all IO while the
>>> filesystem performs occasional IO. If these continue, you need to
>>> discuss with xfs developers how to stop it. If the writes to individual
>>> drives continue after there are no writes to 'md1', then it is worth
>>> coming back here to ask.
>>>
>>>
>> Would the new journal feature be any help?
>
> Probably not, though until we know what is causing the writes, it is
> hard to say.
>
>>
>> I haven't dug in enough to understand it properly, and it would increase
>> the vulnerability of the system to a journal failure, but the feature
>> itself seems almost perfect for batching writes and enabling the disks
>> to spin down for extended periods.
>
> You might be able to build functionality onto the journal which allows
> the drives in the main array to stay idle for longer, but it doesn't try
> to do that at present.
>
> NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-11-11 12:13 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-09 17:20 Question on blocks periodic writes Theophanis Kontogiannis
2016-11-10 2:00 ` NeilBrown
2016-11-10 16:10 ` Wols Lists
2016-11-11 1:52 ` NeilBrown
2016-11-11 12:11 ` Theophanis Kontogiannis
2016-11-11 12:13 ` Theophanis Kontogiannis
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.