From: Yann Dupont <Yann.Dupont@univ-nantes.fr>
To: Yann Dupont <Yann.Dupont@univ-nantes.fr>
Cc: stan@hardwarefreak.com, xfs@oss.sgi.com
Subject: Re: Bad performance with XFS + 2.6.38 / 2.6.39
Date: Thu, 22 Dec 2011 12:02:53 +0100 [thread overview]
Message-ID: <4EF30E5D.7060608@univ-nantes.fr> (raw)
In-Reply-To: <4EF2F702.4050902@univ-nantes.fr>
Le 22/12/2011 10:23, Yann Dupont a écrit :
>
>> Can you run a block trace on both kernels (for say five minutes)
>> when the load differential is showing up and provide that to us so
>> we can see how the IO patterns are differing?
here we go.
1st server : Birnie, is running 2.6.26. This is normally the more loaded
server (more active users)
2nd server : Penderyn, is runing a freshly compiled 3.1.6.
blktrace of relevent volumes during 10 minutes. The 2 machines are
identical (poweredge M1610) : same mem & proc, disks, fibre channel
cards, SAN disks ...
birnie:~/TRACE# uptime
11:48:34 up 17:18, 3 users, load average: 0.04, 0.18, 0.23
penderyn:~/TRACE# uptime
11:48:30 up 23 min, 3 users, load average: 4.03, 3.82, 3.21
As you can see, very sensible load difference. keep in mind my
university is on holiday right now, so the load is really _very much
lower_ than usual. In normal times, with 2.6.26 kernels, birnie has a
load in 2 .. 6 range.
here are the results :
birnie:~/TRACE# blktrace /dev/gromelac/gromelac
/dev/POMEROL-R0-P0/gromeldi -w 600
=== dm-18 ===
CPU 0: 26787 events, 1256 KiB data
CPU 1: 530 events, 25 KiB data
CPU 2: 1811 events, 85 KiB data
CPU 3: 104 events, 5 KiB data
CPU 4: 5824 events, 274 KiB data
CPU 5: 146 events, 7 KiB data
CPU 6: 1958 events, 92 KiB data
CPU 7: 176 events, 9 KiB data
CPU 8: 5456 events, 256 KiB data
CPU 9: 175 events, 9 KiB data
CPU 10: 1161 events, 55 KiB data
CPU 11: 216 events, 11 KiB data
CPU 12: 118 events, 6 KiB data
CPU 13: 25 events, 2 KiB data
CPU 14: 287 events, 14 KiB data
CPU 15: 425 events, 20 KiB data
Total: 45199 events (dropped 0), 2119 KiB data
=== dm-16 ===
CPU 0: 27966 events, 1311 KiB data
CPU 1: 311 events, 15 KiB data
CPU 2: 1403 events, 66 KiB data
CPU 3: 1699 events, 80 KiB data
CPU 4: 1706 events, 80 KiB data
CPU 5: 1515 events, 72 KiB data
CPU 6: 30 events, 2 KiB data
CPU 7: 428 events, 21 KiB data
CPU 8: 6774 events, 318 KiB data
CPU 9: 252 events, 12 KiB data
CPU 10: 1299 events, 61 KiB data
CPU 11: 1391 events, 66 KiB data
CPU 12: 111 events, 6 KiB data
CPU 13: 2317 events, 109 KiB data
CPU 14: 130 events, 7 KiB data
CPU 15: 504 events, 24 KiB data
Total: 47836 events (dropped 0), 2243 KiB data
and
penderyn:~/TRACE# blktrace /dev/gromeljo/gromeljo /dev/gromelpz/gromelpz
/dev/POMEROL-R1-P0/gromelpz -w 600
=== dm-14 ===
CPU 0: 12672 events, 595 KiB data
CPU 1: 13248 events, 621 KiB data
CPU 2: 545 events, 26 KiB data
CPU 3: 285 events, 14 KiB data
CPU 4: 574 events, 27 KiB data
CPU 5: 94 events, 5 KiB data
CPU 6: 569 events, 27 KiB data
CPU 7: 172 events, 9 KiB data
CPU 8: 666 events, 32 KiB data
CPU 9: 3231 events, 152 KiB data
CPU 10: 610 events, 29 KiB data
CPU 11: 221 events, 11 KiB data
CPU 12: 11 events, 1 KiB data
CPU 13: 20 events, 1 KiB data
CPU 14: 6 events, 1 KiB data
CPU 15: 30 events, 2 KiB data
Total: 32954 events (dropped 0), 1545 KiB data
=== dm-13 ===
CPU 0: 0 events, 0 KiB data
CPU 1: 0 events, 0 KiB data
CPU 2: 1 events, 1 KiB data
CPU 3: 0 events, 0 KiB data
CPU 4: 0 events, 0 KiB data
CPU 5: 0 events, 0 KiB data
CPU 6: 0 events, 0 KiB data
CPU 7: 0 events, 0 KiB data
CPU 8: 0 events, 0 KiB data
CPU 9: 0 events, 0 KiB data
CPU 10: 0 events, 0 KiB data
CPU 11: 0 events, 0 KiB data
CPU 12: 0 events, 0 KiB data
CPU 13: 0 events, 0 KiB data
CPU 14: 0 events, 0 KiB data
CPU 15: 0 events, 0 KiB data
Total: 1 events (dropped 0), 1 KiB data
=== dm-16 ===
CPU 0: 17499 events, 821 KiB data
CPU 1: 15320 events, 719 KiB data
CPU 2: 1037 events, 49 KiB data
CPU 3: 667 events, 32 KiB data
CPU 4: 278 events, 14 KiB data
CPU 5: 91 events, 5 KiB data
CPU 6: 888 events, 42 KiB data
CPU 7: 67 events, 4 KiB data
CPU 8: 2317 events, 109 KiB data
CPU 9: 3662 events, 172 KiB data
CPU 10: 1756 events, 83 KiB data
CPU 11: 801 events, 38 KiB data
CPU 12: 20 events, 1 KiB data
CPU 13: 618 events, 29 KiB data
CPU 14: 3 events, 1 KiB data
CPU 15: 18 events, 1 KiB data
Total: 45042 events (dropped 0), 2112 KiB data
And The blktrace files are there (for five days) :
http://filex.univ-nantes.fr/get?k=RDxGitXYOf4HKHd7Tan
Hope it can be helpfull,
Thanks,
--
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2011-12-22 11:05 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-12-11 12:45 Bad performance with XFS + 2.6.38 / 2.6.39 Xupeng Yun
2011-12-11 23:39 ` Dave Chinner
2011-12-12 0:40 ` Xupeng Yun
2011-12-12 1:00 ` Dave Chinner
2011-12-12 2:00 ` Xupeng Yun
2011-12-12 13:57 ` Christoph Hellwig
2011-12-21 9:08 ` Yann Dupont
2011-12-21 15:10 ` Stan Hoeppner
2011-12-21 17:56 ` Yann Dupont
2011-12-21 22:26 ` Dave Chinner
2011-12-22 9:23 ` Yann Dupont
2011-12-22 11:02 ` Yann Dupont [this message]
2012-01-02 10:06 ` Yann Dupont
2012-01-02 16:08 ` Peter Grandi
2012-01-02 18:02 ` Peter Grandi
2012-01-04 10:54 ` Yann Dupont
2012-01-02 20:35 ` Dave Chinner
2012-01-03 8:20 ` Yann Dupont
2012-01-04 12:33 ` Christoph Hellwig
2012-01-04 13:06 ` Yann Dupont
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4EF30E5D.7060608@univ-nantes.fr \
--to=yann.dupont@univ-nantes.fr \
--cc=stan@hardwarefreak.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.