sizing log - is there a too big?

sizing log - is there a too big?

[aurfalien]

Hi all,

Wondering if my log being just under 2GB is a bad idea.

Noticing flush-253:2/kcopyd which is my XFS file system getting really high load avg and wait times via top).

Doing a simple rsync over NFS and after a bit, the system gets to a load of 24.... yikes...

Upon killing the rsync, I am seeing loads going down to sub 1 after about 10 min.  I have repeated this to verify 10 min.

- aurf



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: sizing log - is there a too big?

[Dave Chinner]

On Wed, Jun 26, 2013 at 04:56:31PM -0700, aurfalien wrote:
> Hi all,
> 
> Wondering if my log being just under 2GB is a bad idea.
> 
> Noticing flush-253:2/kcopyd which is my XFS file system getting
> really high load avg and wait times via top).

What has the log size got to do with something that is happening at
the block layer? What's your storage config?

> Doing a simple rsync over NFS and after a bit, the system gets to a load of 24.... yikes...

Let me guess - 24 nfsds blocked waiting for kcopyd to do it's stuff?

Load average going up when the NFS server is busy generally means
your IO subsystem is heavily loaded - it's not uncommon to see large
NFS servers that are extremely busy sustain load averages over a
100 (or even 1000) for hours/days on end....

> Upon killing the rsync, I am seeing loads going down to sub 1
> after about 10 min.  I have repeated this to verify 10 min.

Sure. Processes blocked on IO contribute to the load average. Kill
the IO load, and the load average will return to nothing in 10-15
minutes.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: sizing log - is there a too big?

[aurfalien]

On Jun 26, 2013, at 6:48 PM, Dave Chinner wrote:

> On Wed, Jun 26, 2013 at 04:56:31PM -0700, aurfalien wrote:
>> Hi all,
>> 
>> Wondering if my log being just under 2GB is a bad idea.
>> 
>> Noticing flush-253:2/kcopyd which is my XFS file system getting
>> really high load avg and wait times via top).
> 
> What has the log size got to do with something that is happening at
> the block layer? What's your storage config?
> 
>> Doing a simple rsync over NFS and after a bit, the system gets to a load of 24.... yikes...
> 
> Let me guess - 24 nfsds blocked waiting for kcopyd to do it's stuff?
> 
> Load average going up when the NFS server is busy generally means
> your IO subsystem is heavily loaded - it's not uncommon to see large
> NFS servers that are extremely busy sustain load averages over a
> 100 (or even 1000) for hours/days on end....
> 
>> Upon killing the rsync, I am seeing loads going down to sub 1
>> after about 10 min.  I have repeated this to verify 10 min.
> 
> Sure. Processes blocked on IO contribute to the load average. Kill
> the IO load, and the load average will return to nothing in 10-15
> minutes.


Not so fast my fine feathered friend.

Same work load, same hardware.

Only diff is;

External log, its 2GB
And its Centos 6.4 which was previously 5.9.
Its 16 drives in a hw raid 6, but 2 are for the mirrored journal and 1 for hot spare so 13 spindles.  Before it was 14 spindles in raid 6 and 2 hot spare.

I did power cycle it late last night and will more closely observe today.

Very odd.

- aurf
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: sizing log - is there a too big?

[Dave Chinner]

On Thu, Jun 27, 2013 at 08:58:06AM -0700, aurfalien wrote:
> 
> On Jun 26, 2013, at 6:48 PM, Dave Chinner wrote:
> 
> > On Wed, Jun 26, 2013 at 04:56:31PM -0700, aurfalien wrote:
> >> Hi all,
> >> 
> >> Wondering if my log being just under 2GB is a bad idea.
> >> 
> >> Noticing flush-253:2/kcopyd which is my XFS file system getting
> >> really high load avg and wait times via top).
> > 
> > What has the log size got to do with something that is happening at
> > the block layer? What's your storage config?
> > 
> >> Doing a simple rsync over NFS and after a bit, the system gets to a load of 24.... yikes...
> > 
> > Let me guess - 24 nfsds blocked waiting for kcopyd to do it's stuff?
> > 
> > Load average going up when the NFS server is busy generally means
> > your IO subsystem is heavily loaded - it's not uncommon to see large
> > NFS servers that are extremely busy sustain load averages over a
> > 100 (or even 1000) for hours/days on end....
> > 
> >> Upon killing the rsync, I am seeing loads going down to sub 1
> >> after about 10 min.  I have repeated this to verify 10 min.
> > 
> > Sure. Processes blocked on IO contribute to the load average. Kill
> > the IO load, and the load average will return to nothing in 10-15
> > minutes.
> 
> 
> Not so fast my fine feathered friend.
> 
> Same work load, same hardware.
> 
> Only diff is;
> 
> External log, its 2GB
> And its Centos 6.4 which was previously 5.9.

Oh, you're comparing behaviour between kernels 5 years in age
difference. Well, things change, and a change of load average for
the same workload between very different kernels is no unexpected.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs