* suddenly slow writes on XFS Filesystem
@ 2012-05-06 9:01 Stefan Priebe
2012-05-06 10:31 ` Martin Steigerwald
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Stefan Priebe @ 2012-05-06 9:01 UTC (permalink / raw)
To: xfs
Hi,
since a few days i've experienced a really slow fs on one of our backup
systems.
I'm not sure whether this is XFS related or related to the Controller /
Disks.
It is a raid 10 of 20 SATA Disks and i can only write to them with about
700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4
- no difference. Writing to another partition on another xfs array works
fine.
Details:
#~ df -h
/dev/sdb1 4,6T 4,4T 207G 96% /mnt
#~ df -i
/dev/sdb1 4875737052 4659318044 216419008 96% /mnt
#~ xfs_db -c frag -r /dev/sdb1
actual 83160469, ideal 82145389, fragmentation factor 1,22%
#~ xfs_info /dev/sdb1
meta-data=/dev/sdb1 isize=256 agcount=5,
agsize=268435392 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=1218967031, imaxpct=5
= sunit=64 swidth=1280 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=64 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
#~ cat /proc/mounts
/dev/sdb1 /mnt xfs
rw,noatime,nodiratime,attr2,nobarrier,logbufs=8,logbsize=256k,sunit=512,swidth=10240,prjquota
0 0
Any ideas?
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe
@ 2012-05-06 10:31 ` Martin Steigerwald
2012-05-06 10:33 ` Martin Steigerwald
2012-05-07 1:34 ` Dave Chinner
2 siblings, 0 replies; 33+ messages in thread
From: Martin Steigerwald @ 2012-05-06 10:31 UTC (permalink / raw)
To: xfs; +Cc: Stefan Priebe
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
> Hi,
Hi Stefan,
> since a few days i've experienced a really slow fs on one of our
> backup systems.
>
> I'm not sure whether this is XFS related or related to the Controller
> / Disks.
>
> It is a raid 10 of 20 SATA Disks and i can only write to them with
> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30
> and 3.3.4 - no difference. Writing to another partition on another xfs
> array works fine.
How do you measure the 700 KB/s?
Anything in dmesg?
I and likely others here would be interested in vmstat and iostat outputs
during slowness.
See:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
Ciao,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe
2012-05-06 10:31 ` Martin Steigerwald
@ 2012-05-06 10:33 ` Martin Steigerwald
2012-05-06 15:45 ` Stan Hoeppner
2012-05-07 1:34 ` Dave Chinner
2 siblings, 1 reply; 33+ messages in thread
From: Martin Steigerwald @ 2012-05-06 10:33 UTC (permalink / raw)
To: xfs; +Cc: Stefan Priebe
Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
> It is a raid 10 of 20 SATA Disks and i can only write to them with
> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30
> and 3.3.4 - no difference. Writing to another partition on another xfs
> array works fine.
Additionally what RAID is this? SoftRAID or some - which one? - hardware
RAID controller? And what disks are used, whats the rpm of these?
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 10:33 ` Martin Steigerwald
@ 2012-05-06 15:45 ` Stan Hoeppner
2012-05-06 19:25 ` Stefan Priebe
2012-05-06 21:43 ` Martin Steigerwald
0 siblings, 2 replies; 33+ messages in thread
From: Stan Hoeppner @ 2012-05-06 15:45 UTC (permalink / raw)
To: Martin Steigerwald; +Cc: Stefan Priebe, xfs
On 5/6/2012 5:33 AM, Martin Steigerwald wrote:
> Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
>> It is a raid 10 of 20 SATA Disks and i can only write to them with
>> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30
>> and 3.3.4 - no difference. Writing to another partition on another xfs
>> array works fine.
>
> Additionally what RAID is this? SoftRAID or some - which one? - hardware
> RAID controller? And what disks are used, whats the rpm of these?
I doubt much of this stuff matters. Stefan's filesystem is 96% full,
w/~200GB free. This free space is likely heavily fragmented. If he's
doing allocation in this fragmented free space I'd think that would
fully explain his write performance dropping off a cliff due to massive
head seeking.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 15:45 ` Stan Hoeppner
@ 2012-05-06 19:25 ` Stefan Priebe
2012-05-07 1:39 ` Dave Chinner
2012-05-06 21:43 ` Martin Steigerwald
1 sibling, 1 reply; 33+ messages in thread
From: Stefan Priebe @ 2012-05-06 19:25 UTC (permalink / raw)
To: stan; +Cc: xfs
Am 06.05.2012 17:45, schrieb Stan Hoeppner:
> On 5/6/2012 5:33 AM, Martin Steigerwald wrote:
>> Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
>>> It is a raid 10 of 20 SATA Disks and i can only write to them with
>>> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30
>>> and 3.3.4 - no difference. Writing to another partition on another xfs
>>> array works fine.
>>
>> Additionally what RAID is this? SoftRAID or some - which one? - hardware
>> RAID controller? And what disks are used, whats the rpm of these?
>
> I doubt much of this stuff matters. Stefan's filesystem is 96% full,
> w/~200GB free. This free space is likely heavily fragmented. If he's
> doing allocation in this fragmented free space I'd think that would
> fully explain his write performance dropping off a cliff due to massive
> head seeking.
>
Thanks Stan that's it. After deleting 200GB-300GB it's running fine again.
What is the general recommandation of free space?
Greets Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 15:45 ` Stan Hoeppner
2012-05-06 19:25 ` Stefan Priebe
@ 2012-05-06 21:43 ` Martin Steigerwald
2012-05-07 6:40 ` Stefan Priebe - Profihost AG
1 sibling, 1 reply; 33+ messages in thread
From: Martin Steigerwald @ 2012-05-06 21:43 UTC (permalink / raw)
To: stan; +Cc: Stefan Priebe, xfs
Am Sonntag, 6. Mai 2012 schrieb Stan Hoeppner:
> On 5/6/2012 5:33 AM, Martin Steigerwald wrote:
> > Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
> >> It is a raid 10 of 20 SATA Disks and i can only write to them with
> >> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30
> >> and 3.3.4 - no difference. Writing to another partition on another
> >> xfs array works fine.
> >
> > Additionally what RAID is this? SoftRAID or some - which one? -
> > hardware RAID controller? And what disks are used, whats the rpm of
> > these?
>
> I doubt much of this stuff matters. Stefan's filesystem is 96% full,
> w/~200GB free. This free space is likely heavily fragmented. If he's
> doing allocation in this fragmented free space I'd think that would
> fully explain his write performance dropping off a cliff due to massive
> head seeking.
Hmmm, I thought about that low free space percentage, too, but then 200 GB
free still sounded like quite much to me. Nonetheless, since according to
Stefan freeing some space solved the slowness issue, that seems to have
been it.
Can free space be defragmented in XFS? xfs_fsr only tries to defragment
files as far as I know.
Thanks,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe
2012-05-06 10:31 ` Martin Steigerwald
2012-05-06 10:33 ` Martin Steigerwald
@ 2012-05-07 1:34 ` Dave Chinner
2012-05-07 6:39 ` Stefan Priebe - Profihost AG
2 siblings, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2012-05-07 1:34 UTC (permalink / raw)
To: Stefan Priebe; +Cc: xfs
On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote:
> Hi,
>
> since a few days i've experienced a really slow fs on one of our
> backup systems.
>
> I'm not sure whether this is XFS related or related to the
> Controller / Disks.
>
> It is a raid 10 of 20 SATA Disks and i can only write to them with
> about 700kb/s while doing random i/o.
What sort of random IO? size, read, write, direct or buffered, data
or metadata, etc? iostat -x -d -m 5 and vmstat 5 traces would be
useful to see if it is your array that is slow.....
> I tried vanilla Kernel 3.0.30
> and 3.3.4 - no difference. Writing to another partition on another
> xfs array works fine.
>
> Details:
> #~ df -h
> /dev/sdb1 4,6T 4,4T 207G 96% /mnt
Your filesystem is near full - the allocation algorithms definitely
slow down as you approach ENOSPC, and IO efficiency goes to hell
because of a lack of contiguous free space to allocate from.
> #~ df -i
> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt
You have 4.6 *billion* inodes in your filesystem?
> Any ideas?
None until I understand your workload....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 19:25 ` Stefan Priebe
@ 2012-05-07 1:39 ` Dave Chinner
0 siblings, 0 replies; 33+ messages in thread
From: Dave Chinner @ 2012-05-07 1:39 UTC (permalink / raw)
To: Stefan Priebe; +Cc: stan, xfs
On Sun, May 06, 2012 at 09:25:10PM +0200, Stefan Priebe wrote:
> Am 06.05.2012 17:45, schrieb Stan Hoeppner:
> >On 5/6/2012 5:33 AM, Martin Steigerwald wrote:
> >>Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
> >>>It is a raid 10 of 20 SATA Disks and i can only write to them with
> >>>about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30
> >>>and 3.3.4 - no difference. Writing to another partition on another xfs
> >>>array works fine.
> >>
> >>Additionally what RAID is this? SoftRAID or some - which one? - hardware
> >>RAID controller? And what disks are used, whats the rpm of these?
> >
> >I doubt much of this stuff matters. Stefan's filesystem is 96% full,
> >w/~200GB free. This free space is likely heavily fragmented. If he's
> >doing allocation in this fragmented free space I'd think that would
> >fully explain his write performance dropping off a cliff due to massive
> >head seeking.
> >
> Thanks Stan that's it. After deleting 200GB-300GB it's running fine again.
>
> What is the general recommandation of free space?
Depends on the size of the filesystem. If you've got a 500TB
filesystem, then running at 98% full (10TB of free space) is not
going to be a big deal. But running at 98% full on a 5TB volume is a
big deal because there is relatively little freespace per AG and it
will get rapidly fragmented.
So for a 5TB volume, I' say don't run sustained operations at over
90% full. Going above this temporarily won't be a problem, but
staying at >95% full will definitely cause accelerated aging of the
filesystem.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 1:34 ` Dave Chinner
@ 2012-05-07 6:39 ` Stefan Priebe - Profihost AG
2012-05-07 7:17 ` Dave Chinner
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-05-07 6:39 UTC (permalink / raw)
To: Dave Chinner; +Cc: stan, xfs
Hi,
after deleting 400GB it was faster. Now there are still 300GB free but
it is slow as hell again ;-(
Am 07.05.2012 03:34, schrieb Dave Chinner:
> On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote:
>> Hi,
>>
>> since a few days i've experienced a really slow fs on one of our
>> backup systems.
>>
>> I'm not sure whether this is XFS related or related to the
>> Controller / Disks.
>>
>> It is a raid 10 of 20 SATA Disks and i can only write to them with
>> about 700kb/s while doing random i/o.
>
> What sort of random IO? size, read, write, direct or buffered, data
> or metadata, etc?
There are 4 rsync processes running and doing backups of other severs.
> iostat -x -d -m 5 and vmstat 5 traces would be
> useful to see if it is your array that is slow.....
~ # iostat -x -d -m 5
Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
(8 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 254,80 25,40 1,72 0,16
13,71 0,86 3,08 2,39 67,06
sda 0,00 0,20 0,00 1,20 0,00 0,00
6,50 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 187,40 24,20 1,26 0,19
14,05 0,75 3,56 3,33 70,50
sda 0,00 0,00 0,00 0,40 0,00 0,00
4,50 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 11,20 242,40 92,00 1,56 0,89
15,00 4,70 14,06 1,58 52,68
sda 0,00 0,20 0,00 2,60 0,00 0,02
12,00 0,00 0,00 0,00 0,00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 166,20 24,00 0,99 0,17
12,51 0,57 3,02 2,40 45,56
sda 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00
qDevice: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 188,00 25,40 1,22 0,16
13,23 0,44 2,04 1,78 38,02
sda 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00
# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
7 0 0 788632 48 12189652 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
4 0 0 778148 48 12189776 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
2 0 0 774372 48 12189876 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
5 0 0 771240 48 12189936 0 0 173 395 13 45 1
16 82 1
[root@server844-han /serverbackup (master)]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
6 0 0 768636 48 12190000 0 0 173 395 13 45 1
16 82 1
>
>> I tried vanilla Kernel 3.0.30
>> and 3.3.4 - no difference. Writing to another partition on another
>> xfs array works fine.
>>
>> Details:
>> #~ df -h
>> /dev/sdb1 4,6T 4,4T 207G 96% /mnt
>
> Your filesystem is near full - the allocation algorithms definitely
> slow down as you approach ENOSPC, and IO efficiency goes to hell
> because of a lack of contiguous free space to allocate from.
I've now 94% used but it is still slow. It seems it was just getting
fast with more than 450GB free space.
/dev/sdb1 4,6T 4,3T 310G 94% /mnt
>> #~ df -i
>> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt
> You have 4.6 *billion* inodes in your filesystem?
Yes - it backups around 100 servers with a lot of files.
Greet Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-06 21:43 ` Martin Steigerwald
@ 2012-05-07 6:40 ` Stefan Priebe - Profihost AG
0 siblings, 0 replies; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-05-07 6:40 UTC (permalink / raw)
To: Martin Steigerwald; +Cc: stan, xfs
Am 06.05.2012 23:43, schrieb Martin Steigerwald:
> Am Sonntag, 6. Mai 2012 schrieb Stan Hoeppner:
>> On 5/6/2012 5:33 AM, Martin Steigerwald wrote:
>>> Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe:
>>>> It is a raid 10 of 20 SATA Disks and i can only write to them with
>>>> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30
>>>> and 3.3.4 - no difference. Writing to another partition on another
>>>> xfs array works fine.
>>>
>>> Additionally what RAID is this? SoftRAID or some - which one? -
>>> hardware RAID controller? And what disks are used, whats the rpm of
>>> these?
It is a HW Adaptec 5805 Controller with 20x 7200rpm SATA Disks.
> Can free space be defragmented in XFS? xfs_fsr only tries to defragment
> files as far as I know.
That is a good question.
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 6:39 ` Stefan Priebe - Profihost AG
@ 2012-05-07 7:17 ` Dave Chinner
2012-05-07 7:22 ` Stefan Priebe - Profihost AG
2012-05-07 8:21 ` Martin Steigerwald
2012-05-07 8:31 ` Martin Steigerwald
2 siblings, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2012-05-07 7:17 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: stan, xfs
On Mon, May 07, 2012 at 08:39:13AM +0200, Stefan Priebe - Profihost AG wrote:
> Hi,
>
> after deleting 400GB it was faster. Now there are still 300GB free but
> it is slow as hell again ;-(
>
> Am 07.05.2012 03:34, schrieb Dave Chinner:
> > On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote:
> >> Hi,
> >>
> >> since a few days i've experienced a really slow fs on one of our
> >> backup systems.
> >>
> >> I'm not sure whether this is XFS related or related to the
> >> Controller / Disks.
> >>
> >> It is a raid 10 of 20 SATA Disks and i can only write to them with
> >> about 700kb/s while doing random i/o.
> >
> > What sort of random IO? size, read, write, direct or buffered, data
> > or metadata, etc?
> There are 4 rsync processes running and doing backups of other severs.
>
> > iostat -x -d -m 5 and vmstat 5 traces would be
> > useful to see if it is your array that is slow.....
>
> ~ # iostat -x -d -m 5
> Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
> (8 CPU)
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06
> sda 0,00 0,20 0,00 1,20 0,00 0,00 6,50 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50
> sda 0,00 0,00 0,00 0,40 0,00 0,00 4,50 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 11,20 242,40 92,00 1,56 0,89 15,00 4,70 14,06 1,58 52,68
> sda 0,00 0,20 0,00 2,60 0,00 0,02 12,00 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56
> sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02
> sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
> # vmstat
"vmstat 5", not vmstat 5 times.... :/
> >> I tried vanilla Kernel 3.0.30
> >> and 3.3.4 - no difference. Writing to another partition on another
> >> xfs array works fine.
> >>
> >> Details:
> >> #~ df -h
> >> /dev/sdb1 4,6T 4,4T 207G 96% /mnt
> >
> > Your filesystem is near full - the allocation algorithms definitely
> > slow down as you approach ENOSPC, and IO efficiency goes to hell
> > because of a lack of contiguous free space to allocate from.
> I've now 94% used but it is still slow. It seems it was just getting
> fast with more than 450GB free space.
>
> /dev/sdb1 4,6T 4,3T 310G 94% /mnt
Well, you've probably badly fragmented the free space you have. what
does the 'xfs_db -r -c freesp <dev>' command tell you?
> >> #~ df -i
> >> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt
> > You have 4.6 *billion* inodes in your filesystem?
> Yes - it backups around 100 servers with a lot of files.
So you have what - lots of symlinks? I mean, 4.6 billion inodes
alone requires 1.2TB of space, but if I read the fragmentation
you only have 82 million files with data extents. The only thing
that would other wise use inodes are directories and symlinks....
Still, I can't see how you'd only have 82 million data inodes and 4.5
billion directory inodes - where are all the inodes being consumed?
A massive symlink farm?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 7:17 ` Dave Chinner
@ 2012-05-07 7:22 ` Stefan Priebe - Profihost AG
2012-05-07 16:36 ` Stan Hoeppner
2012-05-07 23:42 ` suddenly slow writes on XFS Filesystem Dave Chinner
0 siblings, 2 replies; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-05-07 7:22 UTC (permalink / raw)
To: Dave Chinner; +Cc: stan, xfs
>> # vmstat
> "vmstat 5", not vmstat 5 times.... :/
oh sorry. Sadly the rsync processes do not run right know i've to kill
them. Is the output still usable?
# vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 1 0 5582136 48 5849956 0 0 176 394 34 54 1
16 82 1
0 1 0 5552180 48 5854280 0 0 2493 2496 3079 2172 1
4 86 9
3 2 0 5601308 48 5857672 0 0 1098 28043 5150 1913 0
10 73 17
0 2 0 5595360 48 5863180 0 0 1098 14336 3945 1897 0
8 69 22
3 2 0 5594088 48 5865280 0 0 432 15897 4209 2366 0
8 71 21
0 2 0 5591068 48 5868940 0 0 854 10989 3519 2107 0
7 70 23
1 1 0 5592004 48 5869872 0 0 180 7886 3605 2436 0
3 76 22
>> /dev/sdb1 4,6T 4,3T 310G 94% /mnt
> Well, you've probably badly fragmented the free space you have. what
> does the 'xfs_db -r -c freesp <dev>' command tell you?
from to extents blocks pct
1 1 942737 942737 0,87
2 3 671860 1590480 1,47
4 7 461268 2416025 2,23
8 15 1350517 18043063 16,67
16 31 111254 2547581 2,35
32 63 192032 9039799 8,35
64 127 33026 3317194 3,07
128 255 14254 2665812 2,46
256 511 12516 4631200 4,28
512 1023 6942 5031081 4,65
1024 2047 4622 6893270 6,37
2048 4095 3268 9412271 8,70
4096 8191 2135 12716435 11,75
8192 16383 338 3974884 3,67
16384 32767 311 7018658 6,49
32768 65535 105 4511372 4,17
65536 131071 29 2577756 2,38
131072 262143 8 1339796 1,24
262144 524287 10 3950416 3,65
524288 1048575 4 2580085 2,38
1048576 2097151 2 3028028 2,80
>>>> #~ df -i
>>>> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt
>>> You have 4.6 *billion* inodes in your filesystem?
>> Yes - it backups around 100 servers with a lot of files.
i rechecked this and it seems i sadly copied the wrong output ;-( sorry
for that.
Here is the correct one:
#~ df -i
/dev/sdb1 975173568 95212355 879961213 10% /mnt
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 6:39 ` Stefan Priebe - Profihost AG
2012-05-07 7:17 ` Dave Chinner
@ 2012-05-07 8:21 ` Martin Steigerwald
2012-05-07 16:44 ` Stan Hoeppner
2012-05-07 8:31 ` Martin Steigerwald
2 siblings, 1 reply; 33+ messages in thread
From: Martin Steigerwald @ 2012-05-07 8:21 UTC (permalink / raw)
To: xfs; +Cc: stan, Stefan Priebe - Profihost AG
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> Hi,
>
> after deleting 400GB it was faster. Now there are still 300GB free but
> it is slow as hell again ;-(
Well, before you had 200 GB free, now you have 300 GB free. Thats not that
much more. You have 5 AGs, so if distributed evenly it would be 40 to 60
GB free space per AG.
The 10% Dave recommended of 4,6 TB would be 460 GB.
Are other volumes of similar size are similarily full but faster?
Would be interesting to see whether you are able to sustain faster speeds
by keeping the volume above 500 GB free or so.
> >> It is a raid 10 of 20 SATA Disks and i can only write to them with
> >> about 700kb/s while doing random i/o.
> >
> > What sort of random IO? size, read, write, direct or buffered, data
> > or metadata, etc?
>
> There are 4 rsync processes running and doing backups of other severs.
Which rsync version is involved? I strongly suggest to use rsync version 3
or above.
> > Your filesystem is near full - the allocation algorithms definitely
> > slow down as you approach ENOSPC, and IO efficiency goes to hell
> > because of a lack of contiguous free space to allocate from.
>
> I've now 94% used but it is still slow. It seems it was just getting
> fast with more than 450GB free space.
>
> /dev/sdb1 4,6T 4,3T 310G 94% /mnt
Then: Will it stay fast if you keep it above 450 GB free space?
And is it as fast then as it used to be before you faced performance
issues?
Ciao,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 6:39 ` Stefan Priebe - Profihost AG
2012-05-07 7:17 ` Dave Chinner
2012-05-07 8:21 ` Martin Steigerwald
@ 2012-05-07 8:31 ` Martin Steigerwald
2012-05-07 13:57 ` Stefan Priebe - Profihost AG
2 siblings, 1 reply; 33+ messages in thread
From: Martin Steigerwald @ 2012-05-07 8:31 UTC (permalink / raw)
To: xfs; +Cc: stan, Stefan Priebe - Profihost AG
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> > iostat -x -d -m 5 and vmstat 5 traces would be
> > useful to see if it is your array that is slow.....
>
> ~ # iostat -x -d -m 5
> Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_
> (8 CPU)
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 254,80 25,40 1,72 0,16
> 13,71 0,86 3,08 2,39 67,06
> sda 0,00 0,20 0,00 1,20 0,00 0,00
> 6,50 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 187,40 24,20 1,26 0,19
> 14,05 0,75 3,56 3,33 70,50
> sda 0,00 0,00 0,00 0,40 0,00 0,00
> 4,50 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 11,20 242,40 92,00 1,56 0,89
> 15,00 4,70 14,06 1,58 52,68
> sda 0,00 0,20 0,00 2,60 0,00 0,02
> 12,00 0,00 0,00 0,00 0,00
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 166,20 24,00 0,99 0,17
> 12,51 0,57 3,02 2,40 45,56
> sda 0,00 0,00 0,00 0,00 0,00 0,00
> 0,00 0,00 0,00 0,00 0,00
>
> qDevice: rrqm/s wrqm/s r/s w/s rMB/s wMB/s
> avgrq-sz avgqu-sz await svctm %util
> sdb 0,00 0,00 188,00 25,40 1,22 0,16
> 13,23 0,44 2,04 1,78 38,02
> sda 0,00 0,00 0,00 0,00 0,00 0,00
> 0,00 0,00 0,00 0,00 0,00
Disk utilization seems to be quite high but it seems there not near to
90 to 100%. So there might be other overheads involved - like network
or (unlikely) CPU.
Did you verify that at the time you perceive slowness the servers you
backup can deliver data fast enough?
I would like to now, whether there are really processes waiting for I/O
during rsync workload.
Can you try vmstat 5 and
while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
while the backup workload is running and slow?
Like this:
merkaba:~> while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2]
root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2]
martin 28374 100 0.0 9800 652 pts/7 D+ 10:27 0:02 dd if=/dev/zero of=fhgs
root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2]
(this is with an Ext4, so its using the flush daemon, with XFS you
probably see xfssyncd or xfsbufd instead if I am not mistaken and if
rsync processes are waiting for I/O they should appear there too)
And yes, its important to have vmstat 5 output during workload is
happening to see the amount of CPU time that the kernel cannot use
for processing cause all processes that are runnable wait for I/O.
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 8:31 ` Martin Steigerwald
@ 2012-05-07 13:57 ` Stefan Priebe - Profihost AG
2012-05-07 14:32 ` Martin Steigerwald
0 siblings, 1 reply; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-05-07 13:57 UTC (permalink / raw)
To: Martin Steigerwald; +Cc: stan, xfs
Am 07.05.2012 10:31, schrieb Martin Steigerwald:
> Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> Did you verify that at the time you perceive slowness the servers you
> backup can deliver data fast enough?
Yes. It works fine to another partition.
> I would like to now, whether there are really processes waiting for I/O
> during rsync workload.
>
> Can you try vmstat 5 and
> while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
Here it is:
# vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 1 0 396688 48 7728136 0 0 229 374 22 32 1 12
85 2
2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1 2
95 2
0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2
2 88 7
0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1 2
96 1
0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1 2
95 2
0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2 2
94 2
2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10
4 87 0
1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15
5 81 0
1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11
4 85 0
# while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync
--daemon
root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
--daemon
root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
--daemon
root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync
--daemon
root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
--daemon
root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
--daemon
root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
--daemon
root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync
--daemon
root 13582 1.7 0.0 74308 13280 ? D 15:50 0:05 rsync
--daemon
root 12493 2.0 0.2 87956 48212 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86420 48156 ? D 15:35 0:25 rsync
--daemon
root 12493 2.0 0.2 86420 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync
--daemon
root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 3378 0.1 0.0 0 0 ? D 13:59 0:09
[flush-8:16]
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync
--daemon
root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync
--daemon
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 13:57 ` Stefan Priebe - Profihost AG
@ 2012-05-07 14:32 ` Martin Steigerwald
0 siblings, 0 replies; 33+ messages in thread
From: Martin Steigerwald @ 2012-05-07 14:32 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: stan, xfs
Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> Am 07.05.2012 10:31, schrieb Martin Steigerwald:
> > Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
> > Did you verify that at the time you perceive slowness the servers you
> > backup can deliver data fast enough?
>
> Yes. It works fine to another partition.
With more free space I suppose?
>
> > I would like to now, whether there are really processes waiting for
> > I/O during rsync workload.
> >
> > Can you try vmstat 5 and
> > while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
>
> Here it is:
> # vmstat 5
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy
> id wa
> 0 1 0 396688 48 7728136 0 0 229 374 22 32 1
> 12 85 2
> 2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1
> 2 95 2
> 0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2
> 2 88 7
> 0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1
> 2 96 1
> 0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1
> 2 95 2
> 0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2
> 2 94 2
> 2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10
> 4 87 0
> 1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15
> 5 81 0
> 1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11
> 4 85 0
Hmm, there are some I/O wait times, but they are lower than I expected.
In the last lines the throughput is higher than in the first lines. Does
that correlate to higher speed in rsync? It may also be excessive writes
due to free space fragmentation, but Dave or someone else might be able to
say whether such a huge difference can be blamed to free space
fragmentation alone.
> # while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done
> root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync
> --daemon
> root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync
> --daemon
> root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
> --daemon
> root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync
> --daemon
> root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync
> --daemon
Still these rsyncs appear to be in uninteruptible sleep quite
consistently. They shouldn´t be in that state when waiting on network
socket readyness.
So this workloads seems to be I/O bound to me.
I suggest trying to keep the volume above 500 GB free and see whether that
works consistently.
Thanks,
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 7:22 ` Stefan Priebe - Profihost AG
@ 2012-05-07 16:36 ` Stan Hoeppner
2012-05-07 19:08 ` Martin Steigerwald
2012-05-07 20:05 ` Stefan Priebe
2012-05-07 23:42 ` suddenly slow writes on XFS Filesystem Dave Chinner
1 sibling, 2 replies; 33+ messages in thread
From: Stan Hoeppner @ 2012-05-07 16:36 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: xfs
On 5/7/2012 2:22 AM, Stefan Priebe - Profihost AG wrote:
> from to extents blocks pct
> 1 1 942737 942737 0,87
> 2 3 671860 1590480 1,47
> 4 7 461268 2416025 2,23
> 8 15 1350517 18043063 16,67
> 16 31 111254 2547581 2,35
> 32 63 192032 9039799 8,35
> 64 127 33026 3317194 3,07
> 128 255 14254 2665812 2,46
> 256 511 12516 4631200 4,28
> 512 1023 6942 5031081 4,65
> 1024 2047 4622 6893270 6,37
> 2048 4095 3268 9412271 8,70
> 4096 8191 2135 12716435 11,75
> 8192 16383 338 3974884 3,67
> 16384 32767 311 7018658 6,49
> 32768 65535 105 4511372 4,17
> 65536 131071 29 2577756 2,38
> 131072 262143 8 1339796 1,24
> 262144 524287 10 3950416 3,65
> 524288 1048575 4 2580085 2,38
> 1048576 2097151 2 3028028 2,80
This shows what I originally suspected. Notice how top heavy this
histogram is. Over half of your free space sits on little islands of
8MB or less. 17% is in islands of 60KB or less. This is heavily
fragmented free space. Contrast this with an XFS from the opposite end
of the aging spectrum that is only 1/3rd full and has seen very few
deletes as it has aged:
from to extents blocks pct
1 1 1028 1028 0.01
2 3 1185 2864 0.02
4 7 1656 9007 0.06
8 15 5232 67674 0.41
16 31 36 775 0.00
32 63 7 292 0.00
64 127 4 350 0.00
128 255 6 1286 0.01
256 511 4 1460 0.01
512 1023 5 4278 0.03
1024 2047 5 6965 0.04
2048 4095 5 12935 0.08
4096 8191 1 8179 0.05
8192 16383 2 29570 0.18
16384 32767 2 60352 0.37
32768 65535 3 148594 0.91
65536 131071 2 195575 1.20
131072 262143 2 420917 2.57
524288 1048575 2 1499689 9.17
4194304 6103694 3 13876795 84.88
Notice how it is very bottom heavy, and that 85% of the free space is in
large islands of 16GB to 24GB.
Stefan, at this point in your filesystem's aging process, it may not
matter how much space you keep freeing up, as your deletion of small
files simply adds more heavily fragmented free space to the pool. It's
the nature of your workload causing this. If you were rsyncing and
deleting 500MB files the story would be much different.
What I would suggest is doing an xfsdump to a filesystem on another LUN
or machine, expand the size of this LUN by 50% or more (I gather this is
an external RAID), format it appropriately, then xfsrestore. This will
eliminate your current free space fragmentation, and the 50% size
increase will delay the next occurrence of this problem. If you can't
expand the LUN, simply do the xfsdump/format/xfsrestore, which will give
you contiguous free space.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 8:21 ` Martin Steigerwald
@ 2012-05-07 16:44 ` Stan Hoeppner
0 siblings, 0 replies; 33+ messages in thread
From: Stan Hoeppner @ 2012-05-07 16:44 UTC (permalink / raw)
To: Martin Steigerwald; +Cc: Stefan Priebe - Profihost AG, xfs
On 5/7/2012 3:21 AM, Martin Steigerwald wrote:
> Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG:
>> after deleting 400GB it was faster. Now there are still 300GB free but
>> it is slow as hell again ;-(
>
> Well, before you had 200 GB free, now you have 300 GB free. Thats not that
> much more. You have 5 AGs, so if distributed evenly it would be 40 to 60
> GB free space per AG.
>
> The 10% Dave recommended of 4,6 TB would be 460 GB.
As I mentioned, increasing the size of the free space pool will only
help for a short duration. Most of the files Stefan is deleting are
small files. Thus he's simply punching out small holes of free space,
which is the problem he already has. Now if he could delete a handful
of large files (10GB+) that are mostly contiguous now, that would help
quite a bit, as the resulting free space wouldn't be horribly fragmented.
So the takeaway is that simply deleting files to get more space isn't
the right way to go. Deleting larger files that are currently
contiguous is the way to go. At least for some temporary breathing
room. The "permanent" solution I described in another reply.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 16:36 ` Stan Hoeppner
@ 2012-05-07 19:08 ` Martin Steigerwald
2012-05-07 20:05 ` Stefan Priebe
1 sibling, 0 replies; 33+ messages in thread
From: Martin Steigerwald @ 2012-05-07 19:08 UTC (permalink / raw)
To: xfs, stan; +Cc: Stefan Priebe - Profihost AG
Am Montag, 7. Mai 2012 schrieb Stan Hoeppner:
> What I would suggest is doing an xfsdump to a filesystem on another LUN
> or machine, expand the size of this LUN by 50% or more (I gather this
> is an external RAID), format it appropriately, then xfsrestore. This
> will eliminate your current free space fragmentation, and the 50% size
> increase will delay the next occurrence of this problem. If you can't
> expand the LUN, simply do the xfsdump/format/xfsrestore, which will
> give you contiguous free space.
Maybe the new dm thin provisioning could help here? But it might just move
the fragmentation from the filesystem to the device mapper layer. I did not
yet check the implementation details of the new thin provisioning target.
--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 16:36 ` Stan Hoeppner
2012-05-07 19:08 ` Martin Steigerwald
@ 2012-05-07 20:05 ` Stefan Priebe
2012-05-09 6:57 ` Stan Hoeppner
1 sibling, 1 reply; 33+ messages in thread
From: Stefan Priebe @ 2012-05-07 20:05 UTC (permalink / raw)
To: stan; +Cc: xfs
Am 07.05.2012 18:36, schrieb Stan Hoeppner:
> This shows what I originally suspected. Notice how top heavy this
> histogram is. Over half of your free space sits on little islands of
> 8MB or less. 17% is in islands of 60KB or less. This is heavily
> fragmented free space. Contrast this with an XFS from the opposite end
> of the aging spectrum that is only 1/3rd full and has seen very few
> deletes as it has aged:
...
>
> Notice how it is very bottom heavy, and that 85% of the free space is in
> large islands of 16GB to 24GB.
This totally makes sense too me. Thanks for this explanation.
> Stefan, at this point in your filesystem's aging process, it may not
> matter how much space you keep freeing up, as your deletion of small
> files simply adds more heavily fragmented free space to the pool. It's
> the nature of your workload causing this.
This makes sense - do you have any idea or solution for this? Are
Filesystems, Block layers or something else which suits this problem /
situation?
> What I would suggest is doing an xfsdump to a filesystem on another LUN
> or machine, expand the size of this LUN by 50% or more (I gather this is
> an external RAID), format it appropriately, then xfsrestore. This will
> eliminate your current free space fragmentation, and the 50% size
> increase will delay the next occurrence of this problem. If you can't
> expand the LUN, simply do the xfsdump/format/xfsrestore, which will give
> you contiguous free space.
But this will only help for a few month or perhaps a year.
Greets,
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 7:22 ` Stefan Priebe - Profihost AG
2012-05-07 16:36 ` Stan Hoeppner
@ 2012-05-07 23:42 ` Dave Chinner
1 sibling, 0 replies; 33+ messages in thread
From: Dave Chinner @ 2012-05-07 23:42 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: stan, xfs
On Mon, May 07, 2012 at 09:22:42AM +0200, Stefan Priebe - Profihost AG wrote:
>
> >> # vmstat
> > "vmstat 5", not vmstat 5 times.... :/
> oh sorry. Sadly the rsync processes do not run right know i've to kill
> them. Is the output still usable?
> # vmstat 5
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy
> id wa
> 0 1 0 5582136 48 5849956 0 0 176 394 34 54 1
> 16 82 1
> 0 1 0 5552180 48 5854280 0 0 2493 2496 3079 2172 1
> 4 86 9
> 3 2 0 5601308 48 5857672 0 0 1098 28043 5150 1913 0
> 10 73 17
> 0 2 0 5595360 48 5863180 0 0 1098 14336 3945 1897 0
> 8 69 22
> 3 2 0 5594088 48 5865280 0 0 432 15897 4209 2366 0
> 8 71 21
> 0 2 0 5591068 48 5868940 0 0 854 10989 3519 2107 0
> 7 70 23
> 1 1 0 5592004 48 5869872 0 0 180 7886 3605 2436 0
> 3 76 22
It tells me that there is still quite an IO load on the system even
when the rsyncs are not running...
> >> /dev/sdb1 4,6T 4,3T 310G 94% /mnt
> > Well, you've probably badly fragmented the free space you have. what
> > does the 'xfs_db -r -c freesp <dev>' command tell you?
>
> from to extents blocks pct
> 1 1 942737 942737 0,87
> 2 3 671860 1590480 1,47
> 4 7 461268 2416025 2,23
> 8 15 1350517 18043063 16,67
> 16 31 111254 2547581 2,35
> 32 63 192032 9039799 8,35
So that's roughly 3.7 million free space extents of 256kB or less
totalling about 32% of the freespace (~100GB). That's pretty badly
fragmented, and given the workload, probably unrecoverable. Dump,
mkfs and restore is probably the only way to unfragment the free
space now, but that would only be a temporary solution if you
continue to run at >90% full. Even if you do keep it at below 90%
full, such a workload will age the filesystem and slowly fragment
free space, but it should take a lot longer to get to this state...
> >>>> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt
> >>> You have 4.6 *billion* inodes in your filesystem?
> >> Yes - it backups around 100 servers with a lot of files.
> i rechecked this and it seems i sadly copied the wrong output ;-( sorry
> for that.
>
> Here is the correct one:
> #~ df -i
> /dev/sdb1 975173568 95212355 879961213 10% /mnt
That makes more sense. :)
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-07 20:05 ` Stefan Priebe
@ 2012-05-09 6:57 ` Stan Hoeppner
2012-05-09 7:04 ` Dave Chinner
0 siblings, 1 reply; 33+ messages in thread
From: Stan Hoeppner @ 2012-05-09 6:57 UTC (permalink / raw)
To: Stefan Priebe; +Cc: xfs
On 5/7/2012 3:05 PM, Stefan Priebe wrote:
> Am 07.05.2012 18:36, schrieb Stan Hoeppner:
>> Stefan, at this point in your filesystem's aging process, it may not
>> matter how much space you keep freeing up, as your deletion of small
>> files simply adds more heavily fragmented free space to the pool. It's
>> the nature of your workload causing this.
> This makes sense - do you have any idea or solution for this? Are
> Filesystems, Block layers or something else which suits this problem /
> situation?
The problem isn't the block layer nor the filesystem. The problem is a
combination of the workload and filling the FS to near capacity.
Any workload that regularly allocates and then deletes large quantities
of small files and fills up the filesystem is going to suffer poor
performance from free space fragmentation as the water in the FS gets
close to the rim of the glass. Two other example workloads are large
mail spools on very busy internet mail servers, and maildir storage on
IMAP/POP servers.
In your case there are two solutions to this problem, the second of
which is also the solution for these mail workloads:
1. Use a backup methodology that writes larger files
2. Give your workload a much larger sandbox to play in
Regarding #1, if you're using rsnapshot your disk space shouldn't be
continuously growing, which is does seem to be. If you're not using
rsnapshot look into it.
Regarding #2 ...
>> What I would suggest is doing an xfsdump to a filesystem on another LUN
>> or machine, expand the size of this LUN by 50% or more (I gather this is
>> an external RAID), format it appropriately, then xfsrestore. This will
>> eliminate your current free space fragmentation, and the 50% size
>> increase will delay the next occurrence of this problem. If you can't
>> expand the LUN, simply do the xfsdump/format/xfsrestore, which will give
>> you contiguous free space.
> But this will only help for a few month or perhaps a year.
So you are saying your backup solution will fill up an additional 2.3TB
in less than a year? In that case I'd say you have dramatically
undersized your backup storage and/or are not using file compression to
your advantage. And you're obviously not using archiving to your
advantage or you'd not have the free space fragmentation issue because
you'd be dealing with much larger files.
So the best solution to your current problem, and one that will save you
disk space and thus $$, is to use a backup solution that makes use of
both tar and gzip/bzip2.
You can't fix this fundamental small file free space fragmentation
problem by tuning/tweaking XFS, or switching to another filesystem, as
again, the problem is the workload, not the block layer or FS.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-09 6:57 ` Stan Hoeppner
@ 2012-05-09 7:04 ` Dave Chinner
2012-05-09 7:36 ` Stefan Priebe - Profihost AG
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Dave Chinner @ 2012-05-09 7:04 UTC (permalink / raw)
To: Stan Hoeppner; +Cc: xfs, Stefan Priebe
On Wed, May 09, 2012 at 01:57:01AM -0500, Stan Hoeppner wrote:
> On 5/7/2012 3:05 PM, Stefan Priebe wrote:
> > Am 07.05.2012 18:36, schrieb Stan Hoeppner:
....
> >> What I would suggest is doing an xfsdump to a filesystem on another LUN
> >> or machine, expand the size of this LUN by 50% or more (I gather this is
> >> an external RAID), format it appropriately, then xfsrestore. This will
> >> eliminate your current free space fragmentation, and the 50% size
> >> increase will delay the next occurrence of this problem. If you can't
> >> expand the LUN, simply do the xfsdump/format/xfsrestore, which will give
> >> you contiguous free space.
> > But this will only help for a few month or perhaps a year.
>
> So you are saying your backup solution will fill up an additional 2.3TB
> in less than a year? In that case I'd say you have dramatically
> undersized your backup storage and/or are not using file compression to
> your advantage. And you're obviously not using archiving to your
> advantage or you'd not have the free space fragmentation issue because
> you'd be dealing with much larger files.
>
> So the best solution to your current problem, and one that will save you
> disk space and thus $$, is to use a backup solution that makes use of
> both tar and gzip/bzip2.
Why use the slow method? xfsdump will be much faster than tar. :)
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-09 7:04 ` Dave Chinner
@ 2012-05-09 7:36 ` Stefan Priebe - Profihost AG
2012-05-09 7:49 ` Stan Hoeppner
2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG
2 siblings, 0 replies; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-05-09 7:36 UTC (permalink / raw)
To: Dave Chinner; +Cc: Stan Hoeppner, xfs
Am 09.05.2012 09:04, schrieb Dave Chinner:
>> So the best solution to your current problem, and one that will save you
>> disk space and thus $$, is to use a backup solution that makes use of
>> both tar and gzip/bzip2.
>
> Why use the slow method? xfsdump will be much faster than tar. :)
I tried to use it in this special case but it is hanging on dumping
directories for 3 hours now or is this normal?
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem
2012-05-09 7:04 ` Dave Chinner
2012-05-09 7:36 ` Stefan Priebe - Profihost AG
@ 2012-05-09 7:49 ` Stan Hoeppner
2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG
2 siblings, 0 replies; 33+ messages in thread
From: Stan Hoeppner @ 2012-05-09 7:49 UTC (permalink / raw)
To: Dave Chinner; +Cc: Stefan Priebe, xfs
On 5/9/2012 2:04 AM, Dave Chinner wrote:
> On Wed, May 09, 2012 at 01:57:01AM -0500, Stan Hoeppner wrote:
>> On 5/7/2012 3:05 PM, Stefan Priebe wrote:
>>> Am 07.05.2012 18:36, schrieb Stan Hoeppner:
> ....
>>>> What I would suggest is doing an xfsdump to a filesystem on another LUN
>>>> or machine, expand the size of this LUN by 50% or more (I gather this is
>>>> an external RAID), format it appropriately, then xfsrestore. This will
>>>> eliminate your current free space fragmentation, and the 50% size
>>>> increase will delay the next occurrence of this problem. If you can't
>>>> expand the LUN, simply do the xfsdump/format/xfsrestore, which will give
>>>> you contiguous free space.
>>> But this will only help for a few month or perhaps a year.
>>
>> So you are saying your backup solution will fill up an additional 2.3TB
>> in less than a year? In that case I'd say you have dramatically
>> undersized your backup storage and/or are not using file compression to
>> your advantage. And you're obviously not using archiving to your
>> advantage or you'd not have the free space fragmentation issue because
>> you'd be dealing with much larger files.
>>
>> So the best solution to your current problem, and one that will save you
>> disk space and thus $$, is to use a backup solution that makes use of
>> both tar and gzip/bzip2.
>
> Why use the slow method? xfsdump will be much faster than tar. :)
Certainly faster. I've assumed Stefan is backing up heterogeneous
systems so xfsdump is probably not an option across the board. My
thinking directly above related to the possibility that his current
software may have integrated functions for tar'ing then compressing at
the directory level. This would save space, cause less free space
fragmentation when old files are deleted, and allow decompressing files
that aren't insanely large in the even a single file needs to be
restored from a compressed tar file. Of course the latter can be
accomplished with xfsdump/restore, but not on heterogeneous systems.
And AFAIK there isn't any nifty FOSS backup software that makes use of
xfsdump/restore, requiring some serious custom scripting. If there
exists such backup software please do share. :)
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* 32bit apps and inode64
2012-05-09 7:04 ` Dave Chinner
2012-05-09 7:36 ` Stefan Priebe - Profihost AG
2012-05-09 7:49 ` Stan Hoeppner
@ 2013-02-15 15:06 ` Stefan Priebe - Profihost AG
2013-02-15 21:46 ` Ben Myers
2 siblings, 1 reply; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-02-15 15:06 UTC (permalink / raw)
To: xfs
Hello list,
i've discovered some problems on a host with a disk > 1TB. We've some
binary 32bit applications which are not able to read some directory
anymore after we've formated and installed the system using vanilla
3.7.7 kernel.
Right now we're using 3.0.61 kernel on this host - so 64bit apps work
well and newly created files get 32bit inode numbers as inode64 is not
the default.
Is there a way to find / get all 64bit inode files / dies and convert
them back to 32bit without a reinstall?
Greets,
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64
2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG
@ 2013-02-15 21:46 ` Ben Myers
2013-02-16 10:24 ` Stefan Priebe - Profihost AG
2013-02-17 8:13 ` Jeff Liu
0 siblings, 2 replies; 33+ messages in thread
From: Ben Myers @ 2013-02-15 21:46 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: xfs
Hi Stefan,
On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote:
> i've discovered some problems on a host with a disk > 1TB. We've some
> binary 32bit applications which are not able to read some directory
> anymore after we've formated and installed the system using vanilla
> 3.7.7 kernel.
>
> Right now we're using 3.0.61 kernel on this host - so 64bit apps work
> well and newly created files get 32bit inode numbers as inode64 is not
> the default.
>
> Is there a way to find / get all 64bit inode files / dies and convert
> them back to 32bit without a reinstall?
On IRIX you could use xfs_reno to renumber those inodes.
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
It isn't in xfsprogs today.
Regards,
Ben
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64
2013-02-15 21:46 ` Ben Myers
@ 2013-02-16 10:24 ` Stefan Priebe - Profihost AG
2013-02-17 21:33 ` Dave Chinner
2013-02-17 8:13 ` Jeff Liu
1 sibling, 1 reply; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-02-16 10:24 UTC (permalink / raw)
To: Ben Myers; +Cc: xfs
I'm on linux. So no recommended way for production systems?
Stefan
Am 15.02.2013 um 22:46 schrieb Ben Myers <bpm@sgi.com>:
> Hi Stefan,
>
> On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote:
>> i've discovered some problems on a host with a disk > 1TB. We've some
>> binary 32bit applications which are not able to read some directory
>> anymore after we've formated and installed the system using vanilla
>> 3.7.7 kernel.
>>
>> Right now we're using 3.0.61 kernel on this host - so 64bit apps work
>> well and newly created files get 32bit inode numbers as inode64 is not
>> the default.
>>
>> Is there a way to find / get all 64bit inode files / dies and convert
>> them back to 32bit without a reinstall?
>
> On IRIX you could use xfs_reno to renumber those inodes.
> http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
>
> xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
> http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
>
> It isn't in xfsprogs today.
>
> Regards,
> Ben
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64
2013-02-15 21:46 ` Ben Myers
2013-02-16 10:24 ` Stefan Priebe - Profihost AG
@ 2013-02-17 8:13 ` Jeff Liu
2013-02-19 19:11 ` Ben Myers
1 sibling, 1 reply; 33+ messages in thread
From: Jeff Liu @ 2013-02-17 8:13 UTC (permalink / raw)
To: Ben Myers; +Cc: xfs, Stefan Priebe - Profihost AG
Hi Ben,
On 02/16/2013 05:46 AM, Ben Myers wrote:
> Hi Stefan,
>
> On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote:
>> i've discovered some problems on a host with a disk > 1TB. We've some
>> binary 32bit applications which are not able to read some directory
>> anymore after we've formated and installed the system using vanilla
>> 3.7.7 kernel.
>>
>> Right now we're using 3.0.61 kernel on this host - so 64bit apps work
>> well and newly created files get 32bit inode numbers as inode64 is not
>> the default.
>>
>> Is there a way to find / get all 64bit inode files / dies and convert
>> them back to 32bit without a reinstall?
>
> On IRIX you could use xfs_reno to renumber those inodes.
> http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
>
> xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
> http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
The old patch set was belong to the infrastructures of online shrinking
support. Recently, I realized that I have made a few mistakes in swap
inodes ioctl(2) implementation when I revisit the old patch set at:
http://oss.sgi.com/archives/xfs/2012-11/msg00414.html
Since we have user request and this function is independent to the
shrinking feature, I'd like to work on it at first if you like it.
Thanks,
-Jeff
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64
2013-02-16 10:24 ` Stefan Priebe - Profihost AG
@ 2013-02-17 21:33 ` Dave Chinner
2013-02-18 8:12 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 33+ messages in thread
From: Dave Chinner @ 2013-02-17 21:33 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Ben Myers, xfs
On Sat, Feb 16, 2013 at 11:24:13AM +0100, Stefan Priebe - Profihost AG wrote:
> I'm on linux. So no recommended way for production systems?
>
> Stefan
>
> Am 15.02.2013 um 22:46 schrieb Ben Myers <bpm@sgi.com>:
>
> > Hi Stefan,
> >
> > On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote:
> >> i've discovered some problems on a host with a disk > 1TB. We've some
> >> binary 32bit applications which are not able to read some directory
> >> anymore after we've formated and installed the system using vanilla
> >> 3.7.7 kernel.
> >>
> >> Right now we're using 3.0.61 kernel on this host - so 64bit apps work
> >> well and newly created files get 32bit inode numbers as inode64 is not
> >> the default.
> >>
> >> Is there a way to find / get all 64bit inode files / dies and convert
> >> them back to 32bit without a reinstall?
> >
> > On IRIX you could use xfs_reno to renumber those inodes.
> > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
> >
> > xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
> > http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
> >
> > It isn't in xfsprogs today.
Simple answer: mount with inode32, run find to print out all the
filenames in the filesystem and their inode number, copy the files
with inodes numbers greater than 32 bit to a temporary file and then
rename them over the top of the original.
That's effectively all xfs_reno does, anyway, just with faster
algorithms (like bulkstat) and a bunch of crash resiliency semantics
wrapped around it....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64
2013-02-17 21:33 ` Dave Chinner
@ 2013-02-18 8:12 ` Stefan Priebe - Profihost AG
2013-02-18 22:06 ` Dave Chinner
0 siblings, 1 reply; 33+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-02-18 8:12 UTC (permalink / raw)
To: Dave Chinner; +Cc: Ben Myers, xfs
Hi,
Am 17.02.2013 22:33, schrieb Dave Chinner:
>>> xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
>>> http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
>>>
>>> It isn't in xfsprogs today.
>
> Simple answer: mount with inode32, run find to print out all the
> filenames in the filesystem and their inode number, copy the files
> with inodes numbers greater than 32 bit to a temporary file and then
> rename them over the top of the original.
Thanks, what about directories? Is renaming / moving using mv enough?
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64
2013-02-18 8:12 ` Stefan Priebe - Profihost AG
@ 2013-02-18 22:06 ` Dave Chinner
0 siblings, 0 replies; 33+ messages in thread
From: Dave Chinner @ 2013-02-18 22:06 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Ben Myers, xfs
On Mon, Feb 18, 2013 at 09:12:03AM +0100, Stefan Priebe - Profihost AG wrote:
> Hi,
>
> Am 17.02.2013 22:33, schrieb Dave Chinner:
>
> >>> xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
> >>> http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
> >>>
> >>> It isn't in xfsprogs today.
> >
> > Simple answer: mount with inode32, run find to print out all the
> > filenames in the filesystem and their inode number, copy the files
> > with inodes numbers greater than 32 bit to a temporary file and then
> > rename them over the top of the original.
>
> Thanks, what about directories? Is renaming / moving using mv enough?
Same thing - create a new temp directory, mv all the entries (which
will be a rename operation, anyway) from the old directory into the
temp directory, rename the temp dir over the top of the old one.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64
2013-02-17 8:13 ` Jeff Liu
@ 2013-02-19 19:11 ` Ben Myers
0 siblings, 0 replies; 33+ messages in thread
From: Ben Myers @ 2013-02-19 19:11 UTC (permalink / raw)
To: Jeff Liu; +Cc: xfs, Stefan Priebe - Profihost AG
Hey Jeff,
On Sun, Feb 17, 2013 at 04:13:28PM +0800, Jeff Liu wrote:
> Hi Ben,
>
> On 02/16/2013 05:46 AM, Ben Myers wrote:
> > Hi Stefan,
> >
> > On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote:
> >> i've discovered some problems on a host with a disk > 1TB. We've some
> >> binary 32bit applications which are not able to read some directory
> >> anymore after we've formated and installed the system using vanilla
> >> 3.7.7 kernel.
> >>
> >> Right now we're using 3.0.61 kernel on this host - so 64bit apps work
> >> well and newly created files get 32bit inode numbers as inode64 is not
> >> the default.
> >>
> >> Is there a way to find / get all 64bit inode files / dies and convert
> >> them back to 32bit without a reinstall?
> >
> > On IRIX you could use xfs_reno to renumber those inodes.
> > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z
> >
> > xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu:
> > http://oss.sgi.com/archives/xfs/2012-11/msg00425.html
> The old patch set was belong to the infrastructures of online shrinking
> support. Recently, I realized that I have made a few mistakes in swap
> inodes ioctl(2) implementation when I revisit the old patch set at:
> http://oss.sgi.com/archives/xfs/2012-11/msg00414.html
>
> Since we have user request and this function is independent to the
> shrinking feature, I'd like to work on it at first if you like it.
That sounds good to me. xfs_reno is a worthwhile feature.
Regards,
Ben
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2013-02-19 19:11 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe
2012-05-06 10:31 ` Martin Steigerwald
2012-05-06 10:33 ` Martin Steigerwald
2012-05-06 15:45 ` Stan Hoeppner
2012-05-06 19:25 ` Stefan Priebe
2012-05-07 1:39 ` Dave Chinner
2012-05-06 21:43 ` Martin Steigerwald
2012-05-07 6:40 ` Stefan Priebe - Profihost AG
2012-05-07 1:34 ` Dave Chinner
2012-05-07 6:39 ` Stefan Priebe - Profihost AG
2012-05-07 7:17 ` Dave Chinner
2012-05-07 7:22 ` Stefan Priebe - Profihost AG
2012-05-07 16:36 ` Stan Hoeppner
2012-05-07 19:08 ` Martin Steigerwald
2012-05-07 20:05 ` Stefan Priebe
2012-05-09 6:57 ` Stan Hoeppner
2012-05-09 7:04 ` Dave Chinner
2012-05-09 7:36 ` Stefan Priebe - Profihost AG
2012-05-09 7:49 ` Stan Hoeppner
2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG
2013-02-15 21:46 ` Ben Myers
2013-02-16 10:24 ` Stefan Priebe - Profihost AG
2013-02-17 21:33 ` Dave Chinner
2013-02-18 8:12 ` Stefan Priebe - Profihost AG
2013-02-18 22:06 ` Dave Chinner
2013-02-17 8:13 ` Jeff Liu
2013-02-19 19:11 ` Ben Myers
2012-05-07 23:42 ` suddenly slow writes on XFS Filesystem Dave Chinner
2012-05-07 8:21 ` Martin Steigerwald
2012-05-07 16:44 ` Stan Hoeppner
2012-05-07 8:31 ` Martin Steigerwald
2012-05-07 13:57 ` Stefan Priebe - Profihost AG
2012-05-07 14:32 ` Martin Steigerwald
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.