* suddenly slow writes on XFS Filesystem
@ 2012-05-06 9:01 Stefan Priebe
2012-05-06 10:31 ` Martin Steigerwald
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Stefan Priebe @ 2012-05-06 9:01 UTC (permalink / raw)
To: xfs
Hi,
since a few days i've experienced a really slow fs on one of our backup
systems.
I'm not sure whether this is XFS related or related to the Controller /
Disks.
It is a raid 10 of 20 SATA Disks and i can only write to them with about
700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 and 3.3.4
- no difference. Writing to another partition on another xfs array works
fine.
Details:
#~ df -h
/dev/sdb1 4,6T 4,4T 207G 96% /mnt
#~ df -i
/dev/sdb1 4875737052 4659318044 216419008 96% /mnt
#~ xfs_db -c frag -r /dev/sdb1
actual 83160469, ideal 82145389, fragmentation factor 1,22%
#~ xfs_info /dev/sdb1
meta-data=/dev/sdb1 isize=256 agcount=5,
agsize=268435392 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=1218967031, imaxpct=5
= sunit=64 swidth=1280 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=64 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
#~ cat /proc/mounts
/dev/sdb1 /mnt xfs
rw,noatime,nodiratime,attr2,nobarrier,logbufs=8,logbsize=256k,sunit=512,swidth=10240,prjquota
0 0
Any ideas?
Stefan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: suddenly slow writes on XFS Filesystem 2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe @ 2012-05-06 10:31 ` Martin Steigerwald 2012-05-06 10:33 ` Martin Steigerwald 2012-05-07 1:34 ` Dave Chinner 2 siblings, 0 replies; 33+ messages in thread From: Martin Steigerwald @ 2012-05-06 10:31 UTC (permalink / raw) To: xfs; +Cc: Stefan Priebe Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe: > Hi, Hi Stefan, > since a few days i've experienced a really slow fs on one of our > backup systems. > > I'm not sure whether this is XFS related or related to the Controller > / Disks. > > It is a raid 10 of 20 SATA Disks and i can only write to them with > about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 > and 3.3.4 - no difference. Writing to another partition on another xfs > array works fine. How do you measure the 700 KB/s? Anything in dmesg? I and likely others here would be interested in vmstat and iostat outputs during slowness. See: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F Ciao, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe 2012-05-06 10:31 ` Martin Steigerwald @ 2012-05-06 10:33 ` Martin Steigerwald 2012-05-06 15:45 ` Stan Hoeppner 2012-05-07 1:34 ` Dave Chinner 2 siblings, 1 reply; 33+ messages in thread From: Martin Steigerwald @ 2012-05-06 10:33 UTC (permalink / raw) To: xfs; +Cc: Stefan Priebe Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe: > It is a raid 10 of 20 SATA Disks and i can only write to them with > about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 > and 3.3.4 - no difference. Writing to another partition on another xfs > array works fine. Additionally what RAID is this? SoftRAID or some - which one? - hardware RAID controller? And what disks are used, whats the rpm of these? -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-06 10:33 ` Martin Steigerwald @ 2012-05-06 15:45 ` Stan Hoeppner 2012-05-06 19:25 ` Stefan Priebe 2012-05-06 21:43 ` Martin Steigerwald 0 siblings, 2 replies; 33+ messages in thread From: Stan Hoeppner @ 2012-05-06 15:45 UTC (permalink / raw) To: Martin Steigerwald; +Cc: Stefan Priebe, xfs On 5/6/2012 5:33 AM, Martin Steigerwald wrote: > Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe: >> It is a raid 10 of 20 SATA Disks and i can only write to them with >> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 >> and 3.3.4 - no difference. Writing to another partition on another xfs >> array works fine. > > Additionally what RAID is this? SoftRAID or some - which one? - hardware > RAID controller? And what disks are used, whats the rpm of these? I doubt much of this stuff matters. Stefan's filesystem is 96% full, w/~200GB free. This free space is likely heavily fragmented. If he's doing allocation in this fragmented free space I'd think that would fully explain his write performance dropping off a cliff due to massive head seeking. -- Stan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-06 15:45 ` Stan Hoeppner @ 2012-05-06 19:25 ` Stefan Priebe 2012-05-07 1:39 ` Dave Chinner 2012-05-06 21:43 ` Martin Steigerwald 1 sibling, 1 reply; 33+ messages in thread From: Stefan Priebe @ 2012-05-06 19:25 UTC (permalink / raw) To: stan; +Cc: xfs Am 06.05.2012 17:45, schrieb Stan Hoeppner: > On 5/6/2012 5:33 AM, Martin Steigerwald wrote: >> Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe: >>> It is a raid 10 of 20 SATA Disks and i can only write to them with >>> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 >>> and 3.3.4 - no difference. Writing to another partition on another xfs >>> array works fine. >> >> Additionally what RAID is this? SoftRAID or some - which one? - hardware >> RAID controller? And what disks are used, whats the rpm of these? > > I doubt much of this stuff matters. Stefan's filesystem is 96% full, > w/~200GB free. This free space is likely heavily fragmented. If he's > doing allocation in this fragmented free space I'd think that would > fully explain his write performance dropping off a cliff due to massive > head seeking. > Thanks Stan that's it. After deleting 200GB-300GB it's running fine again. What is the general recommandation of free space? Greets Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-06 19:25 ` Stefan Priebe @ 2012-05-07 1:39 ` Dave Chinner 0 siblings, 0 replies; 33+ messages in thread From: Dave Chinner @ 2012-05-07 1:39 UTC (permalink / raw) To: Stefan Priebe; +Cc: stan, xfs On Sun, May 06, 2012 at 09:25:10PM +0200, Stefan Priebe wrote: > Am 06.05.2012 17:45, schrieb Stan Hoeppner: > >On 5/6/2012 5:33 AM, Martin Steigerwald wrote: > >>Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe: > >>>It is a raid 10 of 20 SATA Disks and i can only write to them with > >>>about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 > >>>and 3.3.4 - no difference. Writing to another partition on another xfs > >>>array works fine. > >> > >>Additionally what RAID is this? SoftRAID or some - which one? - hardware > >>RAID controller? And what disks are used, whats the rpm of these? > > > >I doubt much of this stuff matters. Stefan's filesystem is 96% full, > >w/~200GB free. This free space is likely heavily fragmented. If he's > >doing allocation in this fragmented free space I'd think that would > >fully explain his write performance dropping off a cliff due to massive > >head seeking. > > > Thanks Stan that's it. After deleting 200GB-300GB it's running fine again. > > What is the general recommandation of free space? Depends on the size of the filesystem. If you've got a 500TB filesystem, then running at 98% full (10TB of free space) is not going to be a big deal. But running at 98% full on a 5TB volume is a big deal because there is relatively little freespace per AG and it will get rapidly fragmented. So for a 5TB volume, I' say don't run sustained operations at over 90% full. Going above this temporarily won't be a problem, but staying at >95% full will definitely cause accelerated aging of the filesystem. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-06 15:45 ` Stan Hoeppner 2012-05-06 19:25 ` Stefan Priebe @ 2012-05-06 21:43 ` Martin Steigerwald 2012-05-07 6:40 ` Stefan Priebe - Profihost AG 1 sibling, 1 reply; 33+ messages in thread From: Martin Steigerwald @ 2012-05-06 21:43 UTC (permalink / raw) To: stan; +Cc: Stefan Priebe, xfs Am Sonntag, 6. Mai 2012 schrieb Stan Hoeppner: > On 5/6/2012 5:33 AM, Martin Steigerwald wrote: > > Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe: > >> It is a raid 10 of 20 SATA Disks and i can only write to them with > >> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 > >> and 3.3.4 - no difference. Writing to another partition on another > >> xfs array works fine. > > > > Additionally what RAID is this? SoftRAID or some - which one? - > > hardware RAID controller? And what disks are used, whats the rpm of > > these? > > I doubt much of this stuff matters. Stefan's filesystem is 96% full, > w/~200GB free. This free space is likely heavily fragmented. If he's > doing allocation in this fragmented free space I'd think that would > fully explain his write performance dropping off a cliff due to massive > head seeking. Hmmm, I thought about that low free space percentage, too, but then 200 GB free still sounded like quite much to me. Nonetheless, since according to Stefan freeing some space solved the slowness issue, that seems to have been it. Can free space be defragmented in XFS? xfs_fsr only tries to defragment files as far as I know. Thanks, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-06 21:43 ` Martin Steigerwald @ 2012-05-07 6:40 ` Stefan Priebe - Profihost AG 0 siblings, 0 replies; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2012-05-07 6:40 UTC (permalink / raw) To: Martin Steigerwald; +Cc: stan, xfs Am 06.05.2012 23:43, schrieb Martin Steigerwald: > Am Sonntag, 6. Mai 2012 schrieb Stan Hoeppner: >> On 5/6/2012 5:33 AM, Martin Steigerwald wrote: >>> Am Sonntag, 6. Mai 2012 schrieb Stefan Priebe: >>>> It is a raid 10 of 20 SATA Disks and i can only write to them with >>>> about 700kb/s while doing random i/o. I tried vanilla Kernel 3.0.30 >>>> and 3.3.4 - no difference. Writing to another partition on another >>>> xfs array works fine. >>> >>> Additionally what RAID is this? SoftRAID or some - which one? - >>> hardware RAID controller? And what disks are used, whats the rpm of >>> these? It is a HW Adaptec 5805 Controller with 20x 7200rpm SATA Disks. > Can free space be defragmented in XFS? xfs_fsr only tries to defragment > files as far as I know. That is a good question. Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe 2012-05-06 10:31 ` Martin Steigerwald 2012-05-06 10:33 ` Martin Steigerwald @ 2012-05-07 1:34 ` Dave Chinner 2012-05-07 6:39 ` Stefan Priebe - Profihost AG 2 siblings, 1 reply; 33+ messages in thread From: Dave Chinner @ 2012-05-07 1:34 UTC (permalink / raw) To: Stefan Priebe; +Cc: xfs On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote: > Hi, > > since a few days i've experienced a really slow fs on one of our > backup systems. > > I'm not sure whether this is XFS related or related to the > Controller / Disks. > > It is a raid 10 of 20 SATA Disks and i can only write to them with > about 700kb/s while doing random i/o. What sort of random IO? size, read, write, direct or buffered, data or metadata, etc? iostat -x -d -m 5 and vmstat 5 traces would be useful to see if it is your array that is slow..... > I tried vanilla Kernel 3.0.30 > and 3.3.4 - no difference. Writing to another partition on another > xfs array works fine. > > Details: > #~ df -h > /dev/sdb1 4,6T 4,4T 207G 96% /mnt Your filesystem is near full - the allocation algorithms definitely slow down as you approach ENOSPC, and IO efficiency goes to hell because of a lack of contiguous free space to allocate from. > #~ df -i > /dev/sdb1 4875737052 4659318044 216419008 96% /mnt You have 4.6 *billion* inodes in your filesystem? > Any ideas? None until I understand your workload.... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 1:34 ` Dave Chinner @ 2012-05-07 6:39 ` Stefan Priebe - Profihost AG 2012-05-07 7:17 ` Dave Chinner ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2012-05-07 6:39 UTC (permalink / raw) To: Dave Chinner; +Cc: stan, xfs Hi, after deleting 400GB it was faster. Now there are still 300GB free but it is slow as hell again ;-( Am 07.05.2012 03:34, schrieb Dave Chinner: > On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote: >> Hi, >> >> since a few days i've experienced a really slow fs on one of our >> backup systems. >> >> I'm not sure whether this is XFS related or related to the >> Controller / Disks. >> >> It is a raid 10 of 20 SATA Disks and i can only write to them with >> about 700kb/s while doing random i/o. > > What sort of random IO? size, read, write, direct or buffered, data > or metadata, etc? There are 4 rsync processes running and doing backups of other severs. > iostat -x -d -m 5 and vmstat 5 traces would be > useful to see if it is your array that is slow..... ~ # iostat -x -d -m 5 Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_ (8 CPU) Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06 sda 0,00 0,20 0,00 1,20 0,00 0,00 6,50 0,00 0,00 0,00 0,00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50 sda 0,00 0,00 0,00 0,40 0,00 0,00 4,50 0,00 0,00 0,00 0,00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 11,20 242,40 92,00 1,56 0,89 15,00 4,70 14,06 1,58 52,68 sda 0,00 0,20 0,00 2,60 0,00 0,02 12,00 0,00 0,00 0,00 0,00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56 sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 qDevice: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02 sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 # vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 7 0 0 788632 48 12189652 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 4 0 0 778148 48 12189776 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 0 774372 48 12189876 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 771240 48 12189936 0 0 173 395 13 45 1 16 82 1 [root@server844-han /serverbackup (master)]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 768636 48 12190000 0 0 173 395 13 45 1 16 82 1 > >> I tried vanilla Kernel 3.0.30 >> and 3.3.4 - no difference. Writing to another partition on another >> xfs array works fine. >> >> Details: >> #~ df -h >> /dev/sdb1 4,6T 4,4T 207G 96% /mnt > > Your filesystem is near full - the allocation algorithms definitely > slow down as you approach ENOSPC, and IO efficiency goes to hell > because of a lack of contiguous free space to allocate from. I've now 94% used but it is still slow. It seems it was just getting fast with more than 450GB free space. /dev/sdb1 4,6T 4,3T 310G 94% /mnt >> #~ df -i >> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt > You have 4.6 *billion* inodes in your filesystem? Yes - it backups around 100 servers with a lot of files. Greet Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 6:39 ` Stefan Priebe - Profihost AG @ 2012-05-07 7:17 ` Dave Chinner 2012-05-07 7:22 ` Stefan Priebe - Profihost AG 2012-05-07 8:21 ` Martin Steigerwald 2012-05-07 8:31 ` Martin Steigerwald 2 siblings, 1 reply; 33+ messages in thread From: Dave Chinner @ 2012-05-07 7:17 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: stan, xfs On Mon, May 07, 2012 at 08:39:13AM +0200, Stefan Priebe - Profihost AG wrote: > Hi, > > after deleting 400GB it was faster. Now there are still 300GB free but > it is slow as hell again ;-( > > Am 07.05.2012 03:34, schrieb Dave Chinner: > > On Sun, May 06, 2012 at 11:01:14AM +0200, Stefan Priebe wrote: > >> Hi, > >> > >> since a few days i've experienced a really slow fs on one of our > >> backup systems. > >> > >> I'm not sure whether this is XFS related or related to the > >> Controller / Disks. > >> > >> It is a raid 10 of 20 SATA Disks and i can only write to them with > >> about 700kb/s while doing random i/o. > > > > What sort of random IO? size, read, write, direct or buffered, data > > or metadata, etc? > There are 4 rsync processes running and doing backups of other severs. > > > iostat -x -d -m 5 and vmstat 5 traces would be > > useful to see if it is your array that is slow..... > > ~ # iostat -x -d -m 5 > Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_ > (8 CPU) > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 254,80 25,40 1,72 0,16 13,71 0,86 3,08 2,39 67,06 > sda 0,00 0,20 0,00 1,20 0,00 0,00 6,50 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 187,40 24,20 1,26 0,19 14,05 0,75 3,56 3,33 70,50 > sda 0,00 0,00 0,00 0,40 0,00 0,00 4,50 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 11,20 242,40 92,00 1,56 0,89 15,00 4,70 14,06 1,58 52,68 > sda 0,00 0,20 0,00 2,60 0,00 0,02 12,00 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 166,20 24,00 0,99 0,17 12,51 0,57 3,02 2,40 45,56 > sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 188,00 25,40 1,22 0,16 13,23 0,44 2,04 1,78 38,02 > sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 > # vmstat "vmstat 5", not vmstat 5 times.... :/ > >> I tried vanilla Kernel 3.0.30 > >> and 3.3.4 - no difference. Writing to another partition on another > >> xfs array works fine. > >> > >> Details: > >> #~ df -h > >> /dev/sdb1 4,6T 4,4T 207G 96% /mnt > > > > Your filesystem is near full - the allocation algorithms definitely > > slow down as you approach ENOSPC, and IO efficiency goes to hell > > because of a lack of contiguous free space to allocate from. > I've now 94% used but it is still slow. It seems it was just getting > fast with more than 450GB free space. > > /dev/sdb1 4,6T 4,3T 310G 94% /mnt Well, you've probably badly fragmented the free space you have. what does the 'xfs_db -r -c freesp <dev>' command tell you? > >> #~ df -i > >> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt > > You have 4.6 *billion* inodes in your filesystem? > Yes - it backups around 100 servers with a lot of files. So you have what - lots of symlinks? I mean, 4.6 billion inodes alone requires 1.2TB of space, but if I read the fragmentation you only have 82 million files with data extents. The only thing that would other wise use inodes are directories and symlinks.... Still, I can't see how you'd only have 82 million data inodes and 4.5 billion directory inodes - where are all the inodes being consumed? A massive symlink farm? Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 7:17 ` Dave Chinner @ 2012-05-07 7:22 ` Stefan Priebe - Profihost AG 2012-05-07 16:36 ` Stan Hoeppner 2012-05-07 23:42 ` suddenly slow writes on XFS Filesystem Dave Chinner 0 siblings, 2 replies; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2012-05-07 7:22 UTC (permalink / raw) To: Dave Chinner; +Cc: stan, xfs >> # vmstat > "vmstat 5", not vmstat 5 times.... :/ oh sorry. Sadly the rsync processes do not run right know i've to kill them. Is the output still usable? # vmstat 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 5582136 48 5849956 0 0 176 394 34 54 1 16 82 1 0 1 0 5552180 48 5854280 0 0 2493 2496 3079 2172 1 4 86 9 3 2 0 5601308 48 5857672 0 0 1098 28043 5150 1913 0 10 73 17 0 2 0 5595360 48 5863180 0 0 1098 14336 3945 1897 0 8 69 22 3 2 0 5594088 48 5865280 0 0 432 15897 4209 2366 0 8 71 21 0 2 0 5591068 48 5868940 0 0 854 10989 3519 2107 0 7 70 23 1 1 0 5592004 48 5869872 0 0 180 7886 3605 2436 0 3 76 22 >> /dev/sdb1 4,6T 4,3T 310G 94% /mnt > Well, you've probably badly fragmented the free space you have. what > does the 'xfs_db -r -c freesp <dev>' command tell you? from to extents blocks pct 1 1 942737 942737 0,87 2 3 671860 1590480 1,47 4 7 461268 2416025 2,23 8 15 1350517 18043063 16,67 16 31 111254 2547581 2,35 32 63 192032 9039799 8,35 64 127 33026 3317194 3,07 128 255 14254 2665812 2,46 256 511 12516 4631200 4,28 512 1023 6942 5031081 4,65 1024 2047 4622 6893270 6,37 2048 4095 3268 9412271 8,70 4096 8191 2135 12716435 11,75 8192 16383 338 3974884 3,67 16384 32767 311 7018658 6,49 32768 65535 105 4511372 4,17 65536 131071 29 2577756 2,38 131072 262143 8 1339796 1,24 262144 524287 10 3950416 3,65 524288 1048575 4 2580085 2,38 1048576 2097151 2 3028028 2,80 >>>> #~ df -i >>>> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt >>> You have 4.6 *billion* inodes in your filesystem? >> Yes - it backups around 100 servers with a lot of files. i rechecked this and it seems i sadly copied the wrong output ;-( sorry for that. Here is the correct one: #~ df -i /dev/sdb1 975173568 95212355 879961213 10% /mnt Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 7:22 ` Stefan Priebe - Profihost AG @ 2012-05-07 16:36 ` Stan Hoeppner 2012-05-07 19:08 ` Martin Steigerwald 2012-05-07 20:05 ` Stefan Priebe 2012-05-07 23:42 ` suddenly slow writes on XFS Filesystem Dave Chinner 1 sibling, 2 replies; 33+ messages in thread From: Stan Hoeppner @ 2012-05-07 16:36 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: xfs On 5/7/2012 2:22 AM, Stefan Priebe - Profihost AG wrote: > from to extents blocks pct > 1 1 942737 942737 0,87 > 2 3 671860 1590480 1,47 > 4 7 461268 2416025 2,23 > 8 15 1350517 18043063 16,67 > 16 31 111254 2547581 2,35 > 32 63 192032 9039799 8,35 > 64 127 33026 3317194 3,07 > 128 255 14254 2665812 2,46 > 256 511 12516 4631200 4,28 > 512 1023 6942 5031081 4,65 > 1024 2047 4622 6893270 6,37 > 2048 4095 3268 9412271 8,70 > 4096 8191 2135 12716435 11,75 > 8192 16383 338 3974884 3,67 > 16384 32767 311 7018658 6,49 > 32768 65535 105 4511372 4,17 > 65536 131071 29 2577756 2,38 > 131072 262143 8 1339796 1,24 > 262144 524287 10 3950416 3,65 > 524288 1048575 4 2580085 2,38 > 1048576 2097151 2 3028028 2,80 This shows what I originally suspected. Notice how top heavy this histogram is. Over half of your free space sits on little islands of 8MB or less. 17% is in islands of 60KB or less. This is heavily fragmented free space. Contrast this with an XFS from the opposite end of the aging spectrum that is only 1/3rd full and has seen very few deletes as it has aged: from to extents blocks pct 1 1 1028 1028 0.01 2 3 1185 2864 0.02 4 7 1656 9007 0.06 8 15 5232 67674 0.41 16 31 36 775 0.00 32 63 7 292 0.00 64 127 4 350 0.00 128 255 6 1286 0.01 256 511 4 1460 0.01 512 1023 5 4278 0.03 1024 2047 5 6965 0.04 2048 4095 5 12935 0.08 4096 8191 1 8179 0.05 8192 16383 2 29570 0.18 16384 32767 2 60352 0.37 32768 65535 3 148594 0.91 65536 131071 2 195575 1.20 131072 262143 2 420917 2.57 524288 1048575 2 1499689 9.17 4194304 6103694 3 13876795 84.88 Notice how it is very bottom heavy, and that 85% of the free space is in large islands of 16GB to 24GB. Stefan, at this point in your filesystem's aging process, it may not matter how much space you keep freeing up, as your deletion of small files simply adds more heavily fragmented free space to the pool. It's the nature of your workload causing this. If you were rsyncing and deleting 500MB files the story would be much different. What I would suggest is doing an xfsdump to a filesystem on another LUN or machine, expand the size of this LUN by 50% or more (I gather this is an external RAID), format it appropriately, then xfsrestore. This will eliminate your current free space fragmentation, and the 50% size increase will delay the next occurrence of this problem. If you can't expand the LUN, simply do the xfsdump/format/xfsrestore, which will give you contiguous free space. -- Stan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 16:36 ` Stan Hoeppner @ 2012-05-07 19:08 ` Martin Steigerwald 2012-05-07 20:05 ` Stefan Priebe 1 sibling, 0 replies; 33+ messages in thread From: Martin Steigerwald @ 2012-05-07 19:08 UTC (permalink / raw) To: xfs, stan; +Cc: Stefan Priebe - Profihost AG Am Montag, 7. Mai 2012 schrieb Stan Hoeppner: > What I would suggest is doing an xfsdump to a filesystem on another LUN > or machine, expand the size of this LUN by 50% or more (I gather this > is an external RAID), format it appropriately, then xfsrestore. This > will eliminate your current free space fragmentation, and the 50% size > increase will delay the next occurrence of this problem. If you can't > expand the LUN, simply do the xfsdump/format/xfsrestore, which will > give you contiguous free space. Maybe the new dm thin provisioning could help here? But it might just move the fragmentation from the filesystem to the device mapper layer. I did not yet check the implementation details of the new thin provisioning target. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 16:36 ` Stan Hoeppner 2012-05-07 19:08 ` Martin Steigerwald @ 2012-05-07 20:05 ` Stefan Priebe 2012-05-09 6:57 ` Stan Hoeppner 1 sibling, 1 reply; 33+ messages in thread From: Stefan Priebe @ 2012-05-07 20:05 UTC (permalink / raw) To: stan; +Cc: xfs Am 07.05.2012 18:36, schrieb Stan Hoeppner: > This shows what I originally suspected. Notice how top heavy this > histogram is. Over half of your free space sits on little islands of > 8MB or less. 17% is in islands of 60KB or less. This is heavily > fragmented free space. Contrast this with an XFS from the opposite end > of the aging spectrum that is only 1/3rd full and has seen very few > deletes as it has aged: ... > > Notice how it is very bottom heavy, and that 85% of the free space is in > large islands of 16GB to 24GB. This totally makes sense too me. Thanks for this explanation. > Stefan, at this point in your filesystem's aging process, it may not > matter how much space you keep freeing up, as your deletion of small > files simply adds more heavily fragmented free space to the pool. It's > the nature of your workload causing this. This makes sense - do you have any idea or solution for this? Are Filesystems, Block layers or something else which suits this problem / situation? > What I would suggest is doing an xfsdump to a filesystem on another LUN > or machine, expand the size of this LUN by 50% or more (I gather this is > an external RAID), format it appropriately, then xfsrestore. This will > eliminate your current free space fragmentation, and the 50% size > increase will delay the next occurrence of this problem. If you can't > expand the LUN, simply do the xfsdump/format/xfsrestore, which will give > you contiguous free space. But this will only help for a few month or perhaps a year. Greets, Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 20:05 ` Stefan Priebe @ 2012-05-09 6:57 ` Stan Hoeppner 2012-05-09 7:04 ` Dave Chinner 0 siblings, 1 reply; 33+ messages in thread From: Stan Hoeppner @ 2012-05-09 6:57 UTC (permalink / raw) To: Stefan Priebe; +Cc: xfs On 5/7/2012 3:05 PM, Stefan Priebe wrote: > Am 07.05.2012 18:36, schrieb Stan Hoeppner: >> Stefan, at this point in your filesystem's aging process, it may not >> matter how much space you keep freeing up, as your deletion of small >> files simply adds more heavily fragmented free space to the pool. It's >> the nature of your workload causing this. > This makes sense - do you have any idea or solution for this? Are > Filesystems, Block layers or something else which suits this problem / > situation? The problem isn't the block layer nor the filesystem. The problem is a combination of the workload and filling the FS to near capacity. Any workload that regularly allocates and then deletes large quantities of small files and fills up the filesystem is going to suffer poor performance from free space fragmentation as the water in the FS gets close to the rim of the glass. Two other example workloads are large mail spools on very busy internet mail servers, and maildir storage on IMAP/POP servers. In your case there are two solutions to this problem, the second of which is also the solution for these mail workloads: 1. Use a backup methodology that writes larger files 2. Give your workload a much larger sandbox to play in Regarding #1, if you're using rsnapshot your disk space shouldn't be continuously growing, which is does seem to be. If you're not using rsnapshot look into it. Regarding #2 ... >> What I would suggest is doing an xfsdump to a filesystem on another LUN >> or machine, expand the size of this LUN by 50% or more (I gather this is >> an external RAID), format it appropriately, then xfsrestore. This will >> eliminate your current free space fragmentation, and the 50% size >> increase will delay the next occurrence of this problem. If you can't >> expand the LUN, simply do the xfsdump/format/xfsrestore, which will give >> you contiguous free space. > But this will only help for a few month or perhaps a year. So you are saying your backup solution will fill up an additional 2.3TB in less than a year? In that case I'd say you have dramatically undersized your backup storage and/or are not using file compression to your advantage. And you're obviously not using archiving to your advantage or you'd not have the free space fragmentation issue because you'd be dealing with much larger files. So the best solution to your current problem, and one that will save you disk space and thus $$, is to use a backup solution that makes use of both tar and gzip/bzip2. You can't fix this fundamental small file free space fragmentation problem by tuning/tweaking XFS, or switching to another filesystem, as again, the problem is the workload, not the block layer or FS. -- Stan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-09 6:57 ` Stan Hoeppner @ 2012-05-09 7:04 ` Dave Chinner 2012-05-09 7:36 ` Stefan Priebe - Profihost AG ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: Dave Chinner @ 2012-05-09 7:04 UTC (permalink / raw) To: Stan Hoeppner; +Cc: xfs, Stefan Priebe On Wed, May 09, 2012 at 01:57:01AM -0500, Stan Hoeppner wrote: > On 5/7/2012 3:05 PM, Stefan Priebe wrote: > > Am 07.05.2012 18:36, schrieb Stan Hoeppner: .... > >> What I would suggest is doing an xfsdump to a filesystem on another LUN > >> or machine, expand the size of this LUN by 50% or more (I gather this is > >> an external RAID), format it appropriately, then xfsrestore. This will > >> eliminate your current free space fragmentation, and the 50% size > >> increase will delay the next occurrence of this problem. If you can't > >> expand the LUN, simply do the xfsdump/format/xfsrestore, which will give > >> you contiguous free space. > > But this will only help for a few month or perhaps a year. > > So you are saying your backup solution will fill up an additional 2.3TB > in less than a year? In that case I'd say you have dramatically > undersized your backup storage and/or are not using file compression to > your advantage. And you're obviously not using archiving to your > advantage or you'd not have the free space fragmentation issue because > you'd be dealing with much larger files. > > So the best solution to your current problem, and one that will save you > disk space and thus $$, is to use a backup solution that makes use of > both tar and gzip/bzip2. Why use the slow method? xfsdump will be much faster than tar. :) Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-09 7:04 ` Dave Chinner @ 2012-05-09 7:36 ` Stefan Priebe - Profihost AG 2012-05-09 7:49 ` Stan Hoeppner 2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG 2 siblings, 0 replies; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2012-05-09 7:36 UTC (permalink / raw) To: Dave Chinner; +Cc: Stan Hoeppner, xfs Am 09.05.2012 09:04, schrieb Dave Chinner: >> So the best solution to your current problem, and one that will save you >> disk space and thus $$, is to use a backup solution that makes use of >> both tar and gzip/bzip2. > > Why use the slow method? xfsdump will be much faster than tar. :) I tried to use it in this special case but it is hanging on dumping directories for 3 hours now or is this normal? Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-09 7:04 ` Dave Chinner 2012-05-09 7:36 ` Stefan Priebe - Profihost AG @ 2012-05-09 7:49 ` Stan Hoeppner 2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG 2 siblings, 0 replies; 33+ messages in thread From: Stan Hoeppner @ 2012-05-09 7:49 UTC (permalink / raw) To: Dave Chinner; +Cc: Stefan Priebe, xfs On 5/9/2012 2:04 AM, Dave Chinner wrote: > On Wed, May 09, 2012 at 01:57:01AM -0500, Stan Hoeppner wrote: >> On 5/7/2012 3:05 PM, Stefan Priebe wrote: >>> Am 07.05.2012 18:36, schrieb Stan Hoeppner: > .... >>>> What I would suggest is doing an xfsdump to a filesystem on another LUN >>>> or machine, expand the size of this LUN by 50% or more (I gather this is >>>> an external RAID), format it appropriately, then xfsrestore. This will >>>> eliminate your current free space fragmentation, and the 50% size >>>> increase will delay the next occurrence of this problem. If you can't >>>> expand the LUN, simply do the xfsdump/format/xfsrestore, which will give >>>> you contiguous free space. >>> But this will only help for a few month or perhaps a year. >> >> So you are saying your backup solution will fill up an additional 2.3TB >> in less than a year? In that case I'd say you have dramatically >> undersized your backup storage and/or are not using file compression to >> your advantage. And you're obviously not using archiving to your >> advantage or you'd not have the free space fragmentation issue because >> you'd be dealing with much larger files. >> >> So the best solution to your current problem, and one that will save you >> disk space and thus $$, is to use a backup solution that makes use of >> both tar and gzip/bzip2. > > Why use the slow method? xfsdump will be much faster than tar. :) Certainly faster. I've assumed Stefan is backing up heterogeneous systems so xfsdump is probably not an option across the board. My thinking directly above related to the possibility that his current software may have integrated functions for tar'ing then compressing at the directory level. This would save space, cause less free space fragmentation when old files are deleted, and allow decompressing files that aren't insanely large in the even a single file needs to be restored from a compressed tar file. Of course the latter can be accomplished with xfsdump/restore, but not on heterogeneous systems. And AFAIK there isn't any nifty FOSS backup software that makes use of xfsdump/restore, requiring some serious custom scripting. If there exists such backup software please do share. :) -- Stan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* 32bit apps and inode64 2012-05-09 7:04 ` Dave Chinner 2012-05-09 7:36 ` Stefan Priebe - Profihost AG 2012-05-09 7:49 ` Stan Hoeppner @ 2013-02-15 15:06 ` Stefan Priebe - Profihost AG 2013-02-15 21:46 ` Ben Myers 2 siblings, 1 reply; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2013-02-15 15:06 UTC (permalink / raw) To: xfs Hello list, i've discovered some problems on a host with a disk > 1TB. We've some binary 32bit applications which are not able to read some directory anymore after we've formated and installed the system using vanilla 3.7.7 kernel. Right now we're using 3.0.61 kernel on this host - so 64bit apps work well and newly created files get 32bit inode numbers as inode64 is not the default. Is there a way to find / get all 64bit inode files / dies and convert them back to 32bit without a reinstall? Greets, Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64 2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG @ 2013-02-15 21:46 ` Ben Myers 2013-02-16 10:24 ` Stefan Priebe - Profihost AG 2013-02-17 8:13 ` Jeff Liu 0 siblings, 2 replies; 33+ messages in thread From: Ben Myers @ 2013-02-15 21:46 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: xfs Hi Stefan, On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote: > i've discovered some problems on a host with a disk > 1TB. We've some > binary 32bit applications which are not able to read some directory > anymore after we've formated and installed the system using vanilla > 3.7.7 kernel. > > Right now we're using 3.0.61 kernel on this host - so 64bit apps work > well and newly created files get 32bit inode numbers as inode64 is not > the default. > > Is there a way to find / get all 64bit inode files / dies and convert > them back to 32bit without a reinstall? On IRIX you could use xfs_reno to renumber those inodes. http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu: http://oss.sgi.com/archives/xfs/2012-11/msg00425.html It isn't in xfsprogs today. Regards, Ben _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64 2013-02-15 21:46 ` Ben Myers @ 2013-02-16 10:24 ` Stefan Priebe - Profihost AG 2013-02-17 21:33 ` Dave Chinner 2013-02-17 8:13 ` Jeff Liu 1 sibling, 1 reply; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2013-02-16 10:24 UTC (permalink / raw) To: Ben Myers; +Cc: xfs I'm on linux. So no recommended way for production systems? Stefan Am 15.02.2013 um 22:46 schrieb Ben Myers <bpm@sgi.com>: > Hi Stefan, > > On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote: >> i've discovered some problems on a host with a disk > 1TB. We've some >> binary 32bit applications which are not able to read some directory >> anymore after we've formated and installed the system using vanilla >> 3.7.7 kernel. >> >> Right now we're using 3.0.61 kernel on this host - so 64bit apps work >> well and newly created files get 32bit inode numbers as inode64 is not >> the default. >> >> Is there a way to find / get all 64bit inode files / dies and convert >> them back to 32bit without a reinstall? > > On IRIX you could use xfs_reno to renumber those inodes. > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z > > xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu: > http://oss.sgi.com/archives/xfs/2012-11/msg00425.html > > It isn't in xfsprogs today. > > Regards, > Ben _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64 2013-02-16 10:24 ` Stefan Priebe - Profihost AG @ 2013-02-17 21:33 ` Dave Chinner 2013-02-18 8:12 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 33+ messages in thread From: Dave Chinner @ 2013-02-17 21:33 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: Ben Myers, xfs On Sat, Feb 16, 2013 at 11:24:13AM +0100, Stefan Priebe - Profihost AG wrote: > I'm on linux. So no recommended way for production systems? > > Stefan > > Am 15.02.2013 um 22:46 schrieb Ben Myers <bpm@sgi.com>: > > > Hi Stefan, > > > > On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote: > >> i've discovered some problems on a host with a disk > 1TB. We've some > >> binary 32bit applications which are not able to read some directory > >> anymore after we've formated and installed the system using vanilla > >> 3.7.7 kernel. > >> > >> Right now we're using 3.0.61 kernel on this host - so 64bit apps work > >> well and newly created files get 32bit inode numbers as inode64 is not > >> the default. > >> > >> Is there a way to find / get all 64bit inode files / dies and convert > >> them back to 32bit without a reinstall? > > > > On IRIX you could use xfs_reno to renumber those inodes. > > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z > > > > xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu: > > http://oss.sgi.com/archives/xfs/2012-11/msg00425.html > > > > It isn't in xfsprogs today. Simple answer: mount with inode32, run find to print out all the filenames in the filesystem and their inode number, copy the files with inodes numbers greater than 32 bit to a temporary file and then rename them over the top of the original. That's effectively all xfs_reno does, anyway, just with faster algorithms (like bulkstat) and a bunch of crash resiliency semantics wrapped around it.... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64 2013-02-17 21:33 ` Dave Chinner @ 2013-02-18 8:12 ` Stefan Priebe - Profihost AG 2013-02-18 22:06 ` Dave Chinner 0 siblings, 1 reply; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2013-02-18 8:12 UTC (permalink / raw) To: Dave Chinner; +Cc: Ben Myers, xfs Hi, Am 17.02.2013 22:33, schrieb Dave Chinner: >>> xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu: >>> http://oss.sgi.com/archives/xfs/2012-11/msg00425.html >>> >>> It isn't in xfsprogs today. > > Simple answer: mount with inode32, run find to print out all the > filenames in the filesystem and their inode number, copy the files > with inodes numbers greater than 32 bit to a temporary file and then > rename them over the top of the original. Thanks, what about directories? Is renaming / moving using mv enough? Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64 2013-02-18 8:12 ` Stefan Priebe - Profihost AG @ 2013-02-18 22:06 ` Dave Chinner 0 siblings, 0 replies; 33+ messages in thread From: Dave Chinner @ 2013-02-18 22:06 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: Ben Myers, xfs On Mon, Feb 18, 2013 at 09:12:03AM +0100, Stefan Priebe - Profihost AG wrote: > Hi, > > Am 17.02.2013 22:33, schrieb Dave Chinner: > > >>> xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu: > >>> http://oss.sgi.com/archives/xfs/2012-11/msg00425.html > >>> > >>> It isn't in xfsprogs today. > > > > Simple answer: mount with inode32, run find to print out all the > > filenames in the filesystem and their inode number, copy the files > > with inodes numbers greater than 32 bit to a temporary file and then > > rename them over the top of the original. > > Thanks, what about directories? Is renaming / moving using mv enough? Same thing - create a new temp directory, mv all the entries (which will be a rename operation, anyway) from the old directory into the temp directory, rename the temp dir over the top of the old one. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64 2013-02-15 21:46 ` Ben Myers 2013-02-16 10:24 ` Stefan Priebe - Profihost AG @ 2013-02-17 8:13 ` Jeff Liu 2013-02-19 19:11 ` Ben Myers 1 sibling, 1 reply; 33+ messages in thread From: Jeff Liu @ 2013-02-17 8:13 UTC (permalink / raw) To: Ben Myers; +Cc: xfs, Stefan Priebe - Profihost AG Hi Ben, On 02/16/2013 05:46 AM, Ben Myers wrote: > Hi Stefan, > > On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote: >> i've discovered some problems on a host with a disk > 1TB. We've some >> binary 32bit applications which are not able to read some directory >> anymore after we've formated and installed the system using vanilla >> 3.7.7 kernel. >> >> Right now we're using 3.0.61 kernel on this host - so 64bit apps work >> well and newly created files get 32bit inode numbers as inode64 is not >> the default. >> >> Is there a way to find / get all 64bit inode files / dies and convert >> them back to 32bit without a reinstall? > > On IRIX you could use xfs_reno to renumber those inodes. > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z > > xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu: > http://oss.sgi.com/archives/xfs/2012-11/msg00425.html The old patch set was belong to the infrastructures of online shrinking support. Recently, I realized that I have made a few mistakes in swap inodes ioctl(2) implementation when I revisit the old patch set at: http://oss.sgi.com/archives/xfs/2012-11/msg00414.html Since we have user request and this function is independent to the shrinking feature, I'd like to work on it at first if you like it. Thanks, -Jeff _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: 32bit apps and inode64 2013-02-17 8:13 ` Jeff Liu @ 2013-02-19 19:11 ` Ben Myers 0 siblings, 0 replies; 33+ messages in thread From: Ben Myers @ 2013-02-19 19:11 UTC (permalink / raw) To: Jeff Liu; +Cc: xfs, Stefan Priebe - Profihost AG Hey Jeff, On Sun, Feb 17, 2013 at 04:13:28PM +0800, Jeff Liu wrote: > Hi Ben, > > On 02/16/2013 05:46 AM, Ben Myers wrote: > > Hi Stefan, > > > > On Fri, Feb 15, 2013 at 04:06:40PM +0100, Stefan Priebe - Profihost AG wrote: > >> i've discovered some problems on a host with a disk > 1TB. We've some > >> binary 32bit applications which are not able to read some directory > >> anymore after we've formated and installed the system using vanilla > >> 3.7.7 kernel. > >> > >> Right now we're using 3.0.61 kernel on this host - so 64bit apps work > >> well and newly created files get 32bit inode numbers as inode64 is not > >> the default. > >> > >> Is there a way to find / get all 64bit inode files / dies and convert > >> them back to 32bit without a reinstall? > > > > On IRIX you could use xfs_reno to renumber those inodes. > > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_reno.z > > > > xfs_reno was ported to linux in '07 and was most recently reposted by Jeff Liu: > > http://oss.sgi.com/archives/xfs/2012-11/msg00425.html > The old patch set was belong to the infrastructures of online shrinking > support. Recently, I realized that I have made a few mistakes in swap > inodes ioctl(2) implementation when I revisit the old patch set at: > http://oss.sgi.com/archives/xfs/2012-11/msg00414.html > > Since we have user request and this function is independent to the > shrinking feature, I'd like to work on it at first if you like it. That sounds good to me. xfs_reno is a worthwhile feature. Regards, Ben _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 7:22 ` Stefan Priebe - Profihost AG 2012-05-07 16:36 ` Stan Hoeppner @ 2012-05-07 23:42 ` Dave Chinner 1 sibling, 0 replies; 33+ messages in thread From: Dave Chinner @ 2012-05-07 23:42 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: stan, xfs On Mon, May 07, 2012 at 09:22:42AM +0200, Stefan Priebe - Profihost AG wrote: > > >> # vmstat > > "vmstat 5", not vmstat 5 times.... :/ > oh sorry. Sadly the rsync processes do not run right know i've to kill > them. Is the output still usable? > # vmstat 5 > procs -----------memory---------- ---swap-- -----io---- -system-- > ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy > id wa > 0 1 0 5582136 48 5849956 0 0 176 394 34 54 1 > 16 82 1 > 0 1 0 5552180 48 5854280 0 0 2493 2496 3079 2172 1 > 4 86 9 > 3 2 0 5601308 48 5857672 0 0 1098 28043 5150 1913 0 > 10 73 17 > 0 2 0 5595360 48 5863180 0 0 1098 14336 3945 1897 0 > 8 69 22 > 3 2 0 5594088 48 5865280 0 0 432 15897 4209 2366 0 > 8 71 21 > 0 2 0 5591068 48 5868940 0 0 854 10989 3519 2107 0 > 7 70 23 > 1 1 0 5592004 48 5869872 0 0 180 7886 3605 2436 0 > 3 76 22 It tells me that there is still quite an IO load on the system even when the rsyncs are not running... > >> /dev/sdb1 4,6T 4,3T 310G 94% /mnt > > Well, you've probably badly fragmented the free space you have. what > > does the 'xfs_db -r -c freesp <dev>' command tell you? > > from to extents blocks pct > 1 1 942737 942737 0,87 > 2 3 671860 1590480 1,47 > 4 7 461268 2416025 2,23 > 8 15 1350517 18043063 16,67 > 16 31 111254 2547581 2,35 > 32 63 192032 9039799 8,35 So that's roughly 3.7 million free space extents of 256kB or less totalling about 32% of the freespace (~100GB). That's pretty badly fragmented, and given the workload, probably unrecoverable. Dump, mkfs and restore is probably the only way to unfragment the free space now, but that would only be a temporary solution if you continue to run at >90% full. Even if you do keep it at below 90% full, such a workload will age the filesystem and slowly fragment free space, but it should take a lot longer to get to this state... > >>>> /dev/sdb1 4875737052 4659318044 216419008 96% /mnt > >>> You have 4.6 *billion* inodes in your filesystem? > >> Yes - it backups around 100 servers with a lot of files. > i rechecked this and it seems i sadly copied the wrong output ;-( sorry > for that. > > Here is the correct one: > #~ df -i > /dev/sdb1 975173568 95212355 879961213 10% /mnt That makes more sense. :) Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 6:39 ` Stefan Priebe - Profihost AG 2012-05-07 7:17 ` Dave Chinner @ 2012-05-07 8:21 ` Martin Steigerwald 2012-05-07 16:44 ` Stan Hoeppner 2012-05-07 8:31 ` Martin Steigerwald 2 siblings, 1 reply; 33+ messages in thread From: Martin Steigerwald @ 2012-05-07 8:21 UTC (permalink / raw) To: xfs; +Cc: stan, Stefan Priebe - Profihost AG Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG: > Hi, > > after deleting 400GB it was faster. Now there are still 300GB free but > it is slow as hell again ;-( Well, before you had 200 GB free, now you have 300 GB free. Thats not that much more. You have 5 AGs, so if distributed evenly it would be 40 to 60 GB free space per AG. The 10% Dave recommended of 4,6 TB would be 460 GB. Are other volumes of similar size are similarily full but faster? Would be interesting to see whether you are able to sustain faster speeds by keeping the volume above 500 GB free or so. > >> It is a raid 10 of 20 SATA Disks and i can only write to them with > >> about 700kb/s while doing random i/o. > > > > What sort of random IO? size, read, write, direct or buffered, data > > or metadata, etc? > > There are 4 rsync processes running and doing backups of other severs. Which rsync version is involved? I strongly suggest to use rsync version 3 or above. > > Your filesystem is near full - the allocation algorithms definitely > > slow down as you approach ENOSPC, and IO efficiency goes to hell > > because of a lack of contiguous free space to allocate from. > > I've now 94% used but it is still slow. It seems it was just getting > fast with more than 450GB free space. > > /dev/sdb1 4,6T 4,3T 310G 94% /mnt Then: Will it stay fast if you keep it above 450 GB free space? And is it as fast then as it used to be before you faced performance issues? Ciao, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 8:21 ` Martin Steigerwald @ 2012-05-07 16:44 ` Stan Hoeppner 0 siblings, 0 replies; 33+ messages in thread From: Stan Hoeppner @ 2012-05-07 16:44 UTC (permalink / raw) To: Martin Steigerwald; +Cc: Stefan Priebe - Profihost AG, xfs On 5/7/2012 3:21 AM, Martin Steigerwald wrote: > Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG: >> after deleting 400GB it was faster. Now there are still 300GB free but >> it is slow as hell again ;-( > > Well, before you had 200 GB free, now you have 300 GB free. Thats not that > much more. You have 5 AGs, so if distributed evenly it would be 40 to 60 > GB free space per AG. > > The 10% Dave recommended of 4,6 TB would be 460 GB. As I mentioned, increasing the size of the free space pool will only help for a short duration. Most of the files Stefan is deleting are small files. Thus he's simply punching out small holes of free space, which is the problem he already has. Now if he could delete a handful of large files (10GB+) that are mostly contiguous now, that would help quite a bit, as the resulting free space wouldn't be horribly fragmented. So the takeaway is that simply deleting files to get more space isn't the right way to go. Deleting larger files that are currently contiguous is the way to go. At least for some temporary breathing room. The "permanent" solution I described in another reply. -- Stan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 6:39 ` Stefan Priebe - Profihost AG 2012-05-07 7:17 ` Dave Chinner 2012-05-07 8:21 ` Martin Steigerwald @ 2012-05-07 8:31 ` Martin Steigerwald 2012-05-07 13:57 ` Stefan Priebe - Profihost AG 2 siblings, 1 reply; 33+ messages in thread From: Martin Steigerwald @ 2012-05-07 8:31 UTC (permalink / raw) To: xfs; +Cc: stan, Stefan Priebe - Profihost AG Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG: > > iostat -x -d -m 5 and vmstat 5 traces would be > > useful to see if it is your array that is slow..... > > ~ # iostat -x -d -m 5 > Linux 2.6.40.28intel (server844-han) 05/07/2012 _x86_64_ > (8 CPU) > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 254,80 25,40 1,72 0,16 > 13,71 0,86 3,08 2,39 67,06 > sda 0,00 0,20 0,00 1,20 0,00 0,00 > 6,50 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 187,40 24,20 1,26 0,19 > 14,05 0,75 3,56 3,33 70,50 > sda 0,00 0,00 0,00 0,40 0,00 0,00 > 4,50 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdb 0,00 11,20 242,40 92,00 1,56 0,89 > 15,00 4,70 14,06 1,58 52,68 > sda 0,00 0,20 0,00 2,60 0,00 0,02 > 12,00 0,00 0,00 0,00 0,00 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 166,20 24,00 0,99 0,17 > 12,51 0,57 3,02 2,40 45,56 > sda 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 > > qDevice: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdb 0,00 0,00 188,00 25,40 1,22 0,16 > 13,23 0,44 2,04 1,78 38,02 > sda 0,00 0,00 0,00 0,00 0,00 0,00 > 0,00 0,00 0,00 0,00 0,00 Disk utilization seems to be quite high but it seems there not near to 90 to 100%. So there might be other overheads involved - like network or (unlikely) CPU. Did you verify that at the time you perceive slowness the servers you backup can deliver data fast enough? I would like to now, whether there are really processes waiting for I/O during rsync workload. Can you try vmstat 5 and while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done while the backup workload is running and slow? Like this: merkaba:~> while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2] root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2] martin 28374 100 0.0 9800 652 pts/7 D+ 10:27 0:02 dd if=/dev/zero of=fhgs root 1508 0.0 0.0 0 0 ? D Mai06 0:00 [flush-253:2] (this is with an Ext4, so its using the flush daemon, with XFS you probably see xfssyncd or xfsbufd instead if I am not mistaken and if rsync processes are waiting for I/O they should appear there too) And yes, its important to have vmstat 5 output during workload is happening to see the amount of CPU time that the kernel cannot use for processing cause all processes that are runnable wait for I/O. -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 8:31 ` Martin Steigerwald @ 2012-05-07 13:57 ` Stefan Priebe - Profihost AG 2012-05-07 14:32 ` Martin Steigerwald 0 siblings, 1 reply; 33+ messages in thread From: Stefan Priebe - Profihost AG @ 2012-05-07 13:57 UTC (permalink / raw) To: Martin Steigerwald; +Cc: stan, xfs Am 07.05.2012 10:31, schrieb Martin Steigerwald: > Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG: > Did you verify that at the time you perceive slowness the servers you > backup can deliver data fast enough? Yes. It works fine to another partition. > I would like to now, whether there are really processes waiting for I/O > during rsync workload. > > Can you try vmstat 5 and > while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done Here it is: # vmstat 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 0 396688 48 7728136 0 0 229 374 22 32 1 12 85 2 2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1 2 95 2 0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2 2 88 7 0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1 2 96 1 0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1 2 95 2 0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2 2 94 2 2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10 4 87 0 1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15 5 81 0 1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11 4 85 0 # while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync --daemon root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync --daemon root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync --daemon root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21720 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86420 48120 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync --daemon root 13582 1.7 0.0 74308 13280 ? D 15:50 0:05 rsync --daemon root 12493 2.0 0.2 87956 48212 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 21980 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86420 48156 ? D 15:35 0:25 rsync --daemon root 12493 2.0 0.2 86420 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22244 ? D 15:50 0:11 rsync --daemon root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22508 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:11 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.5 0.1 76832 22772 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 3378 0.1 0.0 0 0 ? D 13:59 0:09 [flush-8:16] root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon root 12493 2.0 0.2 86676 48180 ? D 15:35 0:25 rsync --daemon root 13581 3.4 0.1 76832 23036 ? D 15:50 0:12 rsync --daemon Stefan _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: suddenly slow writes on XFS Filesystem 2012-05-07 13:57 ` Stefan Priebe - Profihost AG @ 2012-05-07 14:32 ` Martin Steigerwald 0 siblings, 0 replies; 33+ messages in thread From: Martin Steigerwald @ 2012-05-07 14:32 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: stan, xfs Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG: > Am 07.05.2012 10:31, schrieb Martin Steigerwald: > > Am Montag, 7. Mai 2012 schrieb Stefan Priebe - Profihost AG: > > Did you verify that at the time you perceive slowness the servers you > > backup can deliver data fast enough? > > Yes. It works fine to another partition. With more free space I suppose? > > > I would like to now, whether there are really processes waiting for > > I/O during rsync workload. > > > > Can you try vmstat 5 and > > while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done > > Here it is: > # vmstat 5 > procs -----------memory---------- ---swap-- -----io---- -system-- > ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy > id wa > 0 1 0 396688 48 7728136 0 0 229 374 22 32 1 > 12 85 2 > 2 0 0 421744 48 7717348 0 0 3722 1169 9015 8389 1 > 2 95 2 > 0 1 0 405780 48 7751904 0 0 8937 207 10780 9290 2 > 2 88 7 > 0 0 0 486928 48 7733300 0 0 2526 1277 8275 7791 1 > 2 96 1 > 0 0 0 416692 48 7778164 0 0 5046 43750 8548 8141 1 > 2 95 2 > 0 0 0 444968 48 7777792 0 0 8021 1709 9315 8573 2 > 2 94 2 > 2 0 0 357924 48 7946532 0 0 48181 1031 17646 12684 10 > 4 87 0 > 1 0 0 348696 48 8137200 0 0 74366 1056 24362 16775 15 > 5 81 0 > 1 0 0 391552 48 8279000 0 0 54693 1242 19224 13957 11 > 4 85 0 Hmm, there are some I/O wait times, but they are lower than I expected. In the last lines the throughput is higher than in the first lines. Does that correlate to higher speed in rsync? It may also be excessive writes due to free space fragmentation, but Dave or someone else might be able to say whether such a huge difference can be blamed to free space fragmentation alone. > # while true; do ps aux | grep " D" | grep -v grep ; sleep 1; done > root 12493 2.0 0.2 101780 48392 ? D 15:35 0:24 rsync > --daemon > root 13581 3.5 0.1 76832 20828 ? D 15:50 0:10 rsync > --daemon > root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync > --daemon > root 12494 3.9 0.2 128220 44328 ? D 15:35 0:47 rsync > --daemon > root 12493 2.0 0.2 101268 48508 ? D 15:35 0:24 rsync > --daemon Still these rsyncs appear to be in uninteruptible sleep quite consistently. They shouldn´t be in that state when waiting on network socket readyness. So this workloads seems to be I/O bound to me. I suggest trying to keep the volume above 500 GB free and see whether that works consistently. Thanks, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2013-02-19 19:11 UTC | newest] Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-05-06 9:01 suddenly slow writes on XFS Filesystem Stefan Priebe 2012-05-06 10:31 ` Martin Steigerwald 2012-05-06 10:33 ` Martin Steigerwald 2012-05-06 15:45 ` Stan Hoeppner 2012-05-06 19:25 ` Stefan Priebe 2012-05-07 1:39 ` Dave Chinner 2012-05-06 21:43 ` Martin Steigerwald 2012-05-07 6:40 ` Stefan Priebe - Profihost AG 2012-05-07 1:34 ` Dave Chinner 2012-05-07 6:39 ` Stefan Priebe - Profihost AG 2012-05-07 7:17 ` Dave Chinner 2012-05-07 7:22 ` Stefan Priebe - Profihost AG 2012-05-07 16:36 ` Stan Hoeppner 2012-05-07 19:08 ` Martin Steigerwald 2012-05-07 20:05 ` Stefan Priebe 2012-05-09 6:57 ` Stan Hoeppner 2012-05-09 7:04 ` Dave Chinner 2012-05-09 7:36 ` Stefan Priebe - Profihost AG 2012-05-09 7:49 ` Stan Hoeppner 2013-02-15 15:06 ` 32bit apps and inode64 Stefan Priebe - Profihost AG 2013-02-15 21:46 ` Ben Myers 2013-02-16 10:24 ` Stefan Priebe - Profihost AG 2013-02-17 21:33 ` Dave Chinner 2013-02-18 8:12 ` Stefan Priebe - Profihost AG 2013-02-18 22:06 ` Dave Chinner 2013-02-17 8:13 ` Jeff Liu 2013-02-19 19:11 ` Ben Myers 2012-05-07 23:42 ` suddenly slow writes on XFS Filesystem Dave Chinner 2012-05-07 8:21 ` Martin Steigerwald 2012-05-07 16:44 ` Stan Hoeppner 2012-05-07 8:31 ` Martin Steigerwald 2012-05-07 13:57 ` Stefan Priebe - Profihost AG 2012-05-07 14:32 ` Martin Steigerwald
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.