From: Joe Landman <landman@scalableinformatics.com>
To: xfs@oss.sgi.com
Subject: bug: xfs_repair becomes very slow when file system has a large sparse file
Date: Fri, 19 Aug 2011 12:37:05 -0400 [thread overview]
Message-ID: <4E4E9131.2050807@scalableinformatics.com> (raw)
(If you prefer we file this on a bug reporting system, please let me
know where and I'll do this).
Scenario: xfs_repair being run against an about 17TB volume, containing
1 large sparse file. Logical size of 7 PB, actual size, a few hundred GB.
Metadata: Kernel = 2.6.32.41, 2.6.39.4, and others. Xfstools 3.1.5.
Hardware RAID ~17TB LUN. Base OS: Centos 5.6 + updates + updated xfs
tools + our kernels. Using external journal on a different device
What we observe:
Running xfs_repair
xfs_repair -l /dev/md2 -vv /dev/sdd2
the system gets to stage 3 and the first ag. Then it appears to stop.
After an hour or so, we strace it, and we see
pread(...) = 4096
occurring about 2-3 per second. An hour later, its down to 1 per
second. An hour after that, its once every 2 seconds.
Also, somewhere on this disk, someone has created an unfortunately large
file
[root@jr4-2 ~]# ls -alF /data/brick-sdd2/dht/scratch/xyzpdq
total 4652823496
d--------- 2 1232 1000 86 Jun 27 20:31 ./
drwx------ 104 1232 1000 65536 Aug 17 23:53 ../
-rw------- 1 1232 1000 21 Jun 27 09:57 Default.Route
-rw------- 1 1232 1000 250 Jun 27 09:57 Gau-00000.inp
-rw------- 1 1232 1000 0 Jun 27 09:57 Gau-00000.d2e
-rw------- 1 1232 1000 7800416534233088 Jun 27 20:18 Gau-00000.rwf
[root@jr4-2 ~]# ls -ahlF /data/brick-sdd2/dht/scratch/xyzpdq
total 4.4T
d--------- 2 1232 1000 86 Jun 27 20:31 ./
drwx------ 104 1232 1000 64K Aug 17 23:53 ../
-rw------- 1 1232 1000 21 Jun 27 09:57 Default.Route
-rw------- 1 1232 1000 250 Jun 27 09:57 Gau-00000.inp
-rw------- 1 1232 1000 0 Jun 27 09:57 Gau-00000.d2e
-rw------- 1 1232 1000 7.0P Jun 27 20:18 Gau-00000.rwf
This isn't a 7PB file system, its a 100TB file system across 3 machines,
roughly 17TB per brick or OSS. The Gau-00000.rwf is obviously a sparse
file, as could be seen with an ls -alsF
Upon removing that file, the xfs_repair completes within ~10 minutes.
Leaving that file on there, the xfs_repair does not terminate, it just
gets asymptotically slower.
I suspect it is looking for extents which are not there as part of the
repair.
Please let me know if you need more information, or if you would like me
to file this somewhere else for official reportage.
Regards,
Joe
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next reply other threads:[~2011-08-19 16:37 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-08-19 16:37 Joe Landman [this message]
2011-08-20 0:26 ` bug: xfs_repair becomes very slow when file system has a large sparse file Dave Chinner
2011-08-20 0:38 ` Joe Landman
2011-08-20 2:00 ` Dave Chinner
2011-09-07 10:43 ` Christoph Hellwig
2012-01-10 17:32 ` Joe Landman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E4E9131.2050807@scalableinformatics.com \
--to=landman@scalableinformatics.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.