git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Brandon Casey <casey@nrlssc.navy.mil>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Alex Riesen <raa.lkml@gmail.com>,
	Bevan Watkiss <bevan.watkiss@cloakware.com>,
	Git Mailing List <git@vger.kernel.org>
Subject: Re:
Date: Fri, 08 May 2009 18:04:52 -0500	[thread overview]
Message-ID: <twBG1KnSrgPNk7NoVey4mgig1BeAk7e1GHOT90PSV9ZGTs-zCWYdtA@cipher.nrlssc.navy.mil> (raw)
In-Reply-To: <alpine.LFD.2.01.0905081432150.4983@localhost.localdomain>

Linus Torvalds wrote:
> 
> On Fri, 8 May 2009, Brandon Casey wrote:
>> Before (cold cache):
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>>  98.60    6.365501         111     57432           lstat64
>>
>> After (cold cache, no lstat fix, just cache_preload):
>> % time     seconds  usecs/call     calls    errors syscall
>> ------ ----------- ----------- --------- --------- ----------------
>>  90.90   23.717981         413     57432           lstat64
> 
> Yes, interesting. I really smells like it's all fixed performance and 
> there is a single lock around it. That 111us -> 413us increase is very 
> consistent with four cores all serializing on the same lock. So it 
> parallelizes to all four cores, but then will take exactly as long in 
> total.

Makes sense to me.

> Quite frankly, 2.6.9 is so old that I have absolutely _no_ memory of what 
> we used to do back then. Not that I follow NFS all that much even now - I 
> did some of the original page cache and dentry work on the Linux NFS 
> client way back when, but that was when I actually used NFS (and we were 
> converting everything to the page cache).
> 
> I've long since forgotten everything I knew, and I'm just as happy about 
> that. But clearly something is bad, and equally clearly it worked much 
> better for you a couple of months ago. Which does imply that there's 
> probably some centos issues.

In case you're not aware CentOS is just repacked RHEL.  I'm not sure if
centos has the resources for investigating problems.  We also have RHEL
licenses, so hopefully I'll be able to come up with something to submit
to them.

> Can you ask your MIS people if it would be possible to at least _test_ a 
> new kernel? In 2.6.9, I'm quite frankly inclined to just say "it will 
> likely never get fixed unless centos knows what it is", but if you test a 
> more modern kernel and see similar issues, then I'll be intrigued.

I think it's possible.  Just not on this specific machine.  Not sure what
we have lying around multi-processor wise.  Also, it won't happen until
next week since it's late Friday afternoon here.

btw, I've since done some more testing on some centos5.3 boxes we have.
I get similar results (less ancient kernel 2.6.18).  I've also scanned
through the errata announcements that RedHat has released for their
kernel updates.  A few of them involve NFS.  Possibly, whatever RedHat
modified in the 5.X kernel was also backported to the 4.X kernel.

> It's kind of sad, but at the same time, NFS was using the BKL up into 
> 2.6.26 or something like that (about a year ago). And your kernel is 
> based on something _much_ older.
> 
> That said, even with the BKL, NFS should allow all the actual IO to be 
> done in parallel (since the BKL is dropped on scheduling). But it's really 
> wasting a _lot_ of CPU time, and that hurts you enormously, even though 
> the cold-cache case still seems to win, judging by your other email:
>
>> Best without patch: 6.02 (systime 1.57)
>>
>>   0.43user 1.57system 0:06.02elapsed 33%CPU (0avgtext+0avgdata 0maxresident)k
>>   5336inputs+0outputs (12major+15472minor)pagefaults 0swaps
>>
>> Best with patch (preload_cache,lstat reduction): 2.69 (systime 10.47)
>>
>>   0.45user 10.47system 0:02.69elapsed 405%CPU (0avgtext+0avgdata 0maxresident)k
>>   5336inputs+0outputs (12major+13985minor)pagefaults 0swaps
> 
> so there's a _huge_ increase in system time (again), but the change from 
> 33% CPU -> 405% CPU makes up for it and you get lower elapsed times.
> 
> But that 7x increase in system time really is sad. I do suspect it's 
> likely due to spinning on the BKL. And if so, then a modern kernel should 
> fix it.

Thanks, I'll try to test next week.

-brandon

  reply	other threads:[~2009-05-08 23:06 UTC|newest]

Thread overview: 83+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2009-05-07 17:01 (unknown), Bevan Watkiss
2009-05-07 17:13 ` Alex Riesen
2009-05-07 17:26   ` Bevan Watkiss
2009-05-07 18:18     ` Alex Riesen
2009-05-07 18:48       ` Bevan Watkiss
2009-05-07 19:56         ` Björn Steinbrink
2009-05-07 18:56     ` Linus Torvalds
2009-05-07 19:37       ` RE: Bevan Watkiss
2009-05-07 20:07         ` RE: Linus Torvalds
2009-05-07 20:20           ` RE: Linus Torvalds
2009-05-07 20:43             ` Junio C Hamano
2009-05-07 21:33               ` Re: Linus Torvalds
2009-05-07 21:55             ` Linus Torvalds
2009-05-07 22:27               ` RE: david
2009-05-07 22:36                 ` RE: Linus Torvalds
2009-05-07 22:43                   ` RE: david
2009-05-07 23:00                     ` RE: Linus Torvalds
2009-05-07 23:07                       ` RE: david
2009-05-07 23:18                         ` RE: Linus Torvalds
2009-05-07 23:31                           ` RE: david
2009-05-07 23:57                             ` Johan Herland
2009-05-08 16:14                             ` Bevan Watkiss
2009-05-08  8:17               ` Alex Riesen
2009-05-08 14:39                 ` Re: Linus Torvalds
2009-05-08 15:51                   ` Re: Brandon Casey
2009-05-08 16:15                     ` Re: Linus Torvalds
2009-05-08 17:27                       ` Re: Brandon Casey
2009-05-08 17:43                         ` Re: Brandon Casey
2009-05-08 21:49                           ` Re: Linus Torvalds
2009-05-08 23:04                             ` Brandon Casey [this message]
2009-05-09 16:44                               ` Re: Linus Torvalds
2009-05-08 17:44                         ` Re: Linus Torvalds
2009-05-08 16:47               ` 'git checkout' and unlink() calls (was: Re: ) Kjetil Barvik
2009-05-08 17:57                 ` Linus Torvalds
     [not found] <20220301070226.2477769-1-jaydeepjd.8914>
2022-03-06 11:10 ` Jaydeep P Das
2022-03-06 11:22   ` Jaydeep Das
  -- strict thread matches above, loose matches on Subject: below --
2021-08-21 14:40 TECOB270_Ganesh Pawar
2021-08-21 23:52 ` Jeff King
2019-11-15 16:03 Martin Nicolay
2019-11-15 16:29 ` Martin Ågren
2019-11-15 16:37   ` Re: Martin Ågren
2019-08-20 17:23 William Baker
2019-08-20 17:27 ` Yagnatinsky, Mark
2019-03-05 14:57 [GSoC][PATCH v2 3/3] t3600: use helpers to replace test -d/f/e/s <path> Eric Sunshine
2019-03-05 23:38 ` Rohit Ashiwal
2019-01-23 10:50 Christopher Hagler
2019-01-23 14:16 ` Cody Kratzer
2019-01-23 14:25   ` Re: Thomas Braun
2019-01-23 16:00   ` Re: Christopher Hagler
2019-01-23 16:35     ` Randall S. Becker
2019-01-24 17:11   ` Johannes Schindelin
2018-10-08 13:33 Netravnen
2018-10-08 13:34 ` Inderpreet Saini
2018-04-27  0:54 [PATCH v3 2/3] merge: Add merge.renames config setting Ben Peart
2018-04-27 18:19 ` Elijah Newren
2018-04-30 13:11   ` Ben Peart
2018-04-30 16:12     ` Re: Elijah Newren
2018-05-02 14:33       ` Re: Ben Peart
2018-02-27  1:18 Alan Gage
2018-02-27 10:26 ` René Scharfe
2017-11-20 15:10 Viet Nguyen
2017-11-20 20:07 ` Stefan Beller
2017-11-12  2:21 hsed
2017-11-13 18:56 ` Stefan Beller
2017-01-25  0:11 [PATCH 7/7] completion: recognize more long-options Cornelius Weig
2017-01-25  0:21 ` Stefan Beller
2017-01-25  0:43   ` Cornelius Weig
2017-01-25  0:52     ` Re: Stefan Beller
2017-01-25  0:54   ` Re: Linus Torvalds
2017-01-25  1:32     ` Re: Eric Wong
2016-04-11 19:04 (unknown), miwilliams
2016-04-12  4:33 ` Stefan Beller
2015-08-19 19:41 Re: christain147
2015-08-19 11:09 Re: christain147
2015-08-05 12:47 (unknown) Ivan Chernyavsky
2015-08-15  9:19 ` Duy Nguyen
2015-08-17 17:49   ` Re: Junio C Hamano
2015-04-08 20:44 (unknown), Mamta Upadhyay
2015-04-08 21:58 ` Thomas Braun
2015-04-09 11:27   ` Re: Konstantin Khomoutov
     [not found] <CANSxx61FaNp5SBXJ8Y+pWn0eDcunmibKR5g8rttnWGdGwEMHCA@mail.gmail.com>
2015-03-18 20:45 ` Re: Junio C Hamano
2015-03-18 21:06   ` Re: Stefan Beller
2015-03-18 21:17     ` Re: Jeff King
2015-03-18 21:28       ` Re: Jeff King
2015-03-18 21:33         ` Re: Junio C Hamano
2015-03-18 21:45           ` Re: Stefan Beller
2015-03-13  1:34 (unknown) cody.taylor
2015-03-13  2:00 ` Duy Nguyen
2014-09-08 11:36 (unknown), R. Klomp
     [not found] ` <CAOqJoqGSRUw_UT4LhqpYX-WX6AEd2ReAWjgNS76Cra-SMKw3NQ@mail.gmail.com>
2014-09-08 14:36   ` R. Klomp
2014-09-10  0:00     ` Re: David Aguilar
2014-09-15 15:10       ` Re: R. Klomp
2014-02-06 11:54 "Sparse checkout leaves no entry on working directory" all the time on Windows 7 on Git 1.8.5.2.msysgit.0 konstunn
2014-02-06 13:20 ` Johannes Sixt
2014-02-06 19:56   ` Constantine Gorbunov
2012-06-12 21:12 (unknown), rohit sood
2012-06-12 23:51 ` Erik Faye-Lund
2009-11-18  5:03 Re: Anna
2009-05-11 18:57 (unknown) Don Slutz
2009-05-11 20:48 ` Johannes Schindelin
2009-05-12 12:45   ` Re: Don Slutz
2009-03-30  5:03 (unknown), David Aguilar
2009-03-30  7:02 ` Markus Heidelberg
2009-03-30  8:46   ` Re: Junio C Hamano
2007-11-01 20:44 (unknown), Francesco Pretto
2007-11-01 20:48 ` Francesco Pretto
2006-02-02  0:39 [RFC & PATCH] Solaris 8: ENOSYS when mkdir applied to automount., Jason Riedy
2006-02-02  4:18 ` H. Peter Anvin
2005-04-22 22:19 (unknown), atani
2005-04-22 23:16 ` Martin Schlemmer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=twBG1KnSrgPNk7NoVey4mgig1BeAk7e1GHOT90PSV9ZGTs-zCWYdtA@cipher.nrlssc.navy.mil \
    --to=casey@nrlssc.navy.mil \
    --cc=bevan.watkiss@cloakware.com \
    --cc=git@vger.kernel.org \
    --cc=raa.lkml@gmail.com \
    --cc=torvalds@linux-foundation.org \
    --subject='Re:' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).