stripe_cache_active always 0

stripe_cache_active always 0

[Adam Goryachev]

I remember some years ago when I was building my storage servers I spent 
a lot of time optimising performance, and eventually got some really 
good stats and more importantly, good enough performance that users 
stopped complaining, and things "just work".

However, when comparing config with another server, I noticed one of the 
values I configured was stripe_cache_size, and I was setting this to 
4096 (which was the peak performance compared to higher/lower numbers at 
the time).

In any case, I'm looking at a "in use" system, and looking at 
stripe_cache_active and it is always 0. I'm trying to find some docs on 
what this value means?
Is the stripe cache disabled?
Are there just no stripes in the cache?

If it is always 0, then does it mean I don't need to adjust the 
stripe_cache_size (I presume worst case is I am wasting RAM which is not 
really important anyway).

Thanks,
Adam

-- 
Adam Goryachev
Website Managers
P: +61 2 8304 0000                    adam@websitemanagers.com.au
F: +61 2 8304 0001                     www.websitemanagers.com.au

RE: stripe_cache_active always 0

[Robert Kierski]

Adam,

stripe_cache_active is the value to indicate how many pages are currently in use in the stripe cache.  I could be wrong about this... but I don’t think you can disable the stripe cache.  You can shrink it to an unreasonable size, but the minimum you can set stripe_cache_size to is 17 (pages).

Having said that, it may be that your RAID isn't using the stripe cache or that you're not checking the value frequently enough to see stripe cache activity.

A file system that is properly tuned, and doing buffered IO will likely reduce the stripe cache usage.  The stripe cache is basically where data gets put when the RAID is doing partial stripe IO's, calculating parity, or waiting for slow devices to complete their IO's.  It's not intended to be yet another level of cache, so it gets purged when the RAID is done with it.  So, unlike other caches that will hold onto data until there is memory pressure, the stripe cache is more likely to be empty.

As far as adjusting stripe_cache_size... The stripe cache is dynamically allocated.  It won't save any RAM by decreasing stripe_cache_size.  Decreasing it too much will negatively impact performance as it's more likely you'll do Read-Modify-Write's when doing partial stripe writes.

Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax:  651-605-9001
Cell: 651-890-7461

Re: stripe_cache_active always 0

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

On Thu, 7 Jan 2016 16:34:36 +0000
Robert Kierski <rkierski@cray.com> wrote:

> As far as adjusting stripe_cache_size... The stripe cache is dynamically allocated.  It won't save any RAM by decreasing stripe_cache_size.

Since when?

# echo 512 > /sys/devices/virtual/block/md0/md/stripe_cache_size
# free
             total       used       free     shared    buffers     cached
Mem:      16159912   15672696     487216      12588         52   14484708
-/+ buffers/cache:    1187936   14971976
Swap:            0          0          0

# echo 32768 > /sys/devices/virtual/block/md0/md/stripe_cache_size
# free
             total       used       free     shared    buffers     cached
Mem:      16159912   15957880     202032      12588         52   14214952
-/+ buffers/cache:    1742876   14417036
Swap:            0          0          0

You can see that's not the case (on kernel 4.3.3 with four-member RAID5);

And it's quite easy to rapidly hit OOM issues on high-member-count arrays by
setting stripe_cache_size to larger values (not realizing that this is *in
pages* not kilobytes or sectors, and *per disk*).

As for the original question, try checking stripe_cache_active e.g. once
per second during heavy write load to the filesystem.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: {Possible Spam} RE: stripe_cache_active always 0

[Benjamin ESTRABAUD]

On 07/01/16 16:34, Robert Kierski wrote:
> Adam,
>
> stripe_cache_active is the value to indicate how many pages are currently in use in the stripe cache.  I could be wrong about this... but I don’t think you can disable the stripe cache.  You can shrink it to an unreasonable size, but the minimum you can set stripe_cache_size to is 17 (pages).
>
> Having said that, it may be that your RAID isn't using the stripe cache or that you're not checking the value frequently enough to see stripe cache activity.
>
> A file system that is properly tuned, and doing buffered IO will likely reduce the stripe cache usage.  The stripe cache is basically where data gets put when the RAID is doing partial stripe IO's, calculating parity, or waiting for slow devices to complete their IO's.  It's not intended to be yet another level of cache, so it gets purged when the RAID is done with it.  So, unlike other caches that will hold onto data until there is memory pressure, the stripe cache is more likely to be empty.
>
> As far as adjusting stripe_cache_size... The stripe cache is dynamically allocated.  It won't save any RAM by decreasing stripe_cache_size.  Decreasing it too much will negatively impact performance as it's more likely you'll do Read-Modify-Write's when doing partial stripe writes.
>
I find that tuning it a bit higher (but not too high) has a very 
positive impact on rebuild times (where we are therefore building 
stripes). I can't remember exactly the values but we were setting it to 
256 or 512 or even 1024 while building RAID5 or 6 with positive results.

Regards,
Ben.

> Bob Kierski
> Senior Storage Performance Engineer
> Cray Inc.
> 380 Jackson Street
> Suite 210
> St. Paul, MN 55101
> Tele: 651-967-9590
> Fax:  651-605-9001
> Cell: 651-890-7461
>
> N�����r��y���b�X��ǧv�^�)޺{.n�+����{�����{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!tml=
>

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html